Metadata-Version: 2.1
Name: rcd-dev-kit
Version: 2.0.1rc0
Summary: Interact with OIP ecosystem.
Home-page: https://github.com/OpenInnovationProgram/rcd-dev-kit
License: MIT
Author: Davi FACANHA
Requires-Python: >=3.8,<4.0
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.8
Classifier: Topic :: Software Development :: Build Tools
Provides-Extra: dagster
Provides-Extra: docs
Provides-Extra: google
Provides-Extra: py-test
Requires-Dist: Sphinx (>=5.2.3,<6.0.0) ; extra == "docs"
Requires-Dist: boto3 (>=1.22.7,<2.0.0)
Requires-Dist: botocore (>=1.25.7,<2.0.0)
Requires-Dist: connectorx (>=0.3.0,<0.4.0)
Requires-Dist: dagit (>=1.0.16,<2.0.0) ; extra == "dagster"
Requires-Dist: dagster (>=1.0.16,<2.0.0) ; extra == "dagster"
Requires-Dist: dagster-k8s (==0.16.16) ; extra == "dagster"
Requires-Dist: dagster-postgres (==0.16.16) ; extra == "dagster"
Requires-Dist: elasticsearch (>=7.0.0,<8.0.0)
Requires-Dist: google (>=3.0.0,<4.0.0) ; extra == "google"
Requires-Dist: google-api-python-client (>=2.47.0,<3.0.0) ; extra == "google"
Requires-Dist: google-cloud-storage (>=2.3.0,<3.0.0) ; extra == "google"
Requires-Dist: googletrans (==3.0.0) ; extra == "google"
Requires-Dist: myst-nb (>=0.17.1,<0.18.0) ; (python_version >= "3.8" and python_version < "4.0") and (extra == "docs")
Requires-Dist: numpy (>=1.22.3,<2.0.0)
Requires-Dist: pandas (>=1.4.2,<2.0.0)
Requires-Dist: psycopg2-binary (>=2.9.3,<3.0.0)
Requires-Dist: py-markdown-table (>=0.3.3,<0.4.0)
Requires-Dist: pyspark (>=3.0.0,<4.0.0)
Requires-Dist: pytest (>=7.1,<8.0) ; extra == "py-test"
Requires-Dist: pytest-cov (>=3.0.0,<4.0.0) ; extra == "py-test"
Requires-Dist: python-dotenv (>=0.20.0,<0.21.0)
Requires-Dist: requests (>=2.28.1,<3.0.0) ; extra == "dagster"
Requires-Dist: scikit-learn (>=1.1.2,<2.0.0)
Requires-Dist: snowflake-connector-python[pandas] (==2.7.9)
Requires-Dist: sphinx-autoapi (>=2.0.0,<3.0.0) ; extra == "docs"
Requires-Dist: sphinx-rtd-theme (==1.0.0) ; extra == "docs"
Requires-Dist: sphinxcontrib-napoleon (==0.7) ; extra == "docs"
Requires-Dist: sqlalchemy (>=1.4,<2.0)
Requires-Dist: sqlalchemy-redshift (>=0.8.11,<0.9.0)
Requires-Dist: sqlparse (>=0.4.2,<0.5.0)
Requires-Dist: statsmodels (>=0.13.2,<0.14.0)
Requires-Dist: tqdm (>=4.64.1,<5.0.0)
Description-Content-Type: text/markdown

# rcd_dev_kit
### Developed by Real Consulting Data

## Description
We've developed `rcd-dev-kit` to facilitate the manipulation and interaction with the OIP ecosystem.

## Installation
```bash
pip install rcd-dev-kit
```

## Modules
We've divided our functions in four main modules:
- [database_manager](./src/rcd_dev_kit/database_manager)
    - Classes:
        - [GcloudOperator()](./src/rcd_dev_kit/database_manager/gcloud_operator.py)
        - MysqlOperator()
        - ElasticsearchOperator()
        - RedshiftOperator()
        - SnowflakeOperator()
        - S3Operator()
    - Main Functions:
        - index_json_bulk()
        - index_json()
        - index_json_bulk_parallel()
        - [send_to_redshift()](./src/rcd_dev_kit/database_manager/redshift_operator.py)
        - read_from_redshift()
        - [send_metadata_to_redshift()](./src/rcd_dev_kit/database_manager/redshift_operator.py)
        - find_tables_by_column_name()
        - migrate_metadata_from_redshift()
        - upload_raw_s3()
        - download_raw_s3()
        - [upload_to_gcloud()](./src/rcd_dev_kit/database_manager/gcloud_operator.py)
        - [download_from_gcloud()](./src/rcd_dev_kit/database_manager/gcloud_operator.py)

- [dataclass_manager](./src/rcd_dev_kit/dataclass_manager)
    - Classes:
        - RawDataFile()

- [decorator_manager](./src/rcd_dev_kit/decorator_manager)
    - Main Functions:
        - timeit()
        - debug()

- [file_manager](./src/rcd_dev_kit/file_manager)
    - Classes:
        - FileOperator()
        - [FileDownloader()](./src/rcd_dev_kit/file_manager/file_downloader.py)
    - Main Functions:
        - detect_path()
        - detect_all_files()
        - write_df_to_json_parallel()
        - download_excel()
        - download_csv()
        - download_pdf()
        - download_zip()

- [pandas_manager](./src/rcd_dev_kit/pandas_manager)
    - Main Functions:
        - strip_all_text_column()
        - check_na()
        - check_duplication()
        - check_quality_table_names()
        - normalize_date_column()
        - detect_aws_type()

- [sql_utils](./src/rcd_dev_kit/sql_utils)
    - Main Functions:
        - convert_to_snowflake_syntax()
        - correct_sql_system_variables_syntax()

## Pre-requirements
Since some of the functions deal with database connections(S3, Redshift, Snowflake, GCP, Elasticsearch, ...), we must
be careful to sensitive information. Thus, to use the functions correctly we must have a `.env` file following
the `.env.example` template.

## Feedback
Any questions or suggestions?
Please contact package maintainer.

# python-sdk
Refer to book https://py-pkgs.org/01-introduction for best practices

# Maintainers
This package is using poetry for pkg management, it must be installed locally if you are maintaining the package.  
For developing and test the pkg locally, you must run `poetry install`.

**This git repository has an automated CI/CD process** found on the git worflow: [main.yml](.github/workflows/main.yml). It means that once all modifications have been made, a Pull Request to main will trigger a serie of actions:    
- Install Package: `poetry install`
- Run Unitary Tests: `poetry run pytest -v tests/ --cov=rcd_dev_kit --cov-report=xml`
- Build Package: `poetry build`
- Publish Package in PyPI: `poetry publish`
- Install Package from PyPI: `pip install rcd_dev_kit`
- Send a Teams message with the new available version: Git Image `toko-bifrost/ms-teams-deploy-card@master`.

## Contributing
Interested in contributing? Check out the contributing guidelines. Please note that this project is released with a Code of Conduct. By contributing to this project, you agree to abide by its terms.

## License
`rcd_dev_kit` was created by RCD. It is licensed under the terms of the MIT license.

## Credits
`rcd_dev_kit` was created with [`cookiecutter`](https://cookiecutter.readthedocs.io/en/latest/) and the `py-pkgs-cookiecutter` [template](https://github.com/py-pkgs/py-pkgs-cookiecutter).
