Hi

Hello @Armando Ytriago !!!

👋

Welcome @André Ventura and @Tariq Hasan!! I've invited you to this channel as you have expressed interest in contributing. 🙂

Thanks @Thomas Chung!

A new PR to add Apache Druid Data source if anyone interested to review and feedback 😄

Hello @DANGerous I had a quick question regarding the ticket https://github.com/mage-ai/mage-ai/issues/1166. I notice that you mention there that Instead of hardcoding, we should be able to interpolate any variable values using the handlebar syntax: {{ var('env') }} where env is the key we want to extract from the global variables. I was wondering if you require the variable to be entered as a global variable and read in through

global_vars

or if you require the variable to be entered as an environment variable and read in through the

EnvironmentVariableLoader

. I was also curious about the naming for this variable. Would you like it to capitalized as

ENV

similar to how the environment variables are treated when using the handlebar syntax or kept as

env

similar to how the global variables are named? Lastly, I wanted to clarify if the syntax should be

{{ env_var('env') }}

or

{{ var('env') }}

(assuming lower-case convention here).

Hello team 👋 I wanted to reach out and suggest that it would be great to have code documentation available, even if there is already documentation provided separately. Specifically, having documentation directly within the source code, especially for abstractions, would be incredibly helpful for contributors.

Thanks for suggestion! We'll add more documentation to the code to make it easier to understand.

Hello mates 👋 Out of curiosity, when working on Mage, what IDE do you use and what extensions do you find most useful for your development work?

I think there’s some room to improve the contributing docs. When I was trying to set up my IDE for development on mage (in particular on the mage_integrations directory), I was struggling to infer how tests should be run (which runner, and from where). The contributing docs point out a few scripts to run to execute tests, linting etc. I think it would be helpful to add more to the docs (looking at you - https://docs.mage.ai/contributing/backend/overview). Also, a lot of the scripts and setup are coupled to actually running the docker images. What’s the appetite around accepting changes to the docs, testing, dev tools (like pre-commit) and directory structure layout? Is there a process I should go through, or just make changes and try to PR it in?

Hey there, fellow community members! I am Dhanush from India and I am absolutely pumped to be a part of this incredible community! Currently, I am pursuing my Pre-final year in UG - Computer Science and Engineering, and I am super passionate about Python, ML, problem-solving, and front-end development including desktop and mobile applications.

I looking to contribute to this project but this is my first open-source contribution, so I request your help and guidance to get started to contribute as I couldn't able to find any good first issue

Can someone point me to contribute some area with less complexity to get used to this code base

Can we arrange a high level codebase walkthrough session for the community willing to contribute?

Hello, my friends. I'm going to start studying how the mage works his magic. I would love to contribute to the project through code or by promoting the tool here in Brazil. Let's go!

I’m trying to test out some of my changes for implementing the Couchbase destination #2452, but I’m running into an error. I’m trying to create the dev setup by following the instructions found here. When I was trying to run

./scripts/init.sh mage_dev

The docker build was failing on the

pip install -r requirements.txt

. In particular, it was failing to build

uamqp

, which it says is required to install

pyproject.toml

based projects. I tried removing the

pyproject.toml

and

poetry.lock

files in the root of

mage-ai

and in

mage-integrations

, but I still get the same error. Is anyone else having this error? I can’t tell if it’s because my mac is the M2 on the ARM architecture, so it’s failing at the compile step.

I also had some questions around contributing to the codebase. @DANGerous, let me know if I should hold off until the developer success engineer joins full time. • for the

Destination

base class, there are some methods that throw

Exceptions

, saying that derived classes need to implement (

test_connection

and

export_batch_data

); I was wondering if maybe

@abstractmethod

should be used here instead - the reason being communication; if i’m trying to add a new destination, it’s not obvious to me what methods need to be overridden; I had to control F for

Exception('Subclasses

• it was difficult to navigate to <https://docs.mage.ai/data-integrations/destinations/add-new-destination> - I kept trying to navigate to <https://docs.mage.ai/data-integrations/sources/add-new-destination> from the page <https://docs.mage.ai/contributing/backend/overview>, but it was routing me to <https://docs.mage.ai/introduction/overview> (I was only able to easily access that link because I had bookmarked it) • how are dependencies for

mage_integrations

being handled? is it the one big

requirements.txt

that everyone

pip

installs when building the docker image? are there plans for using extras, so if someone is working on a pipeline that’s just using S3 and Snowflake, they don’t need bigquery, mysql and Couchbase SDKs? • is there any support within Mage for asynchronous data reading/writing? And how does the interface for the base

Connection

and

Destination

classes support those? • how are

upsert

or write/read-many patterns handled? SQL-based destinations require you to implement

build_insert_commands

, and NoSQL-like destinations push the responsibility down to the user to pass a valid query string - just looking for some guidance here about how the supported workflows and access patterns are managed for different connections and destinations: ◦ some SDKs support asynch, bulk, read/write-many, in addition to transactional CRUD - I wonder if we want to surface those optimized capabilities to users ◦ I know there’s batch and streaming workflows, but I suppose I’m just looking for more clear guidance in the docs for what contract connectors should adhere to here Thanks!

Where can I propose new integrations?

(that I'd like to work on ..)

_S_hout out to @Ethan Brown for his first contribution! 🥳

I have a quick question about the

requirements.txt

in the

mage_integrations

folder. I see that some of the code in

mage_integrations

depends on

pandas

, but I don’t see

pandas

inside the

requirements.txt

in the

mage_integrations

folder. Does the code in

mage_integrations

use the same virtual environment created from the

requirements.txt

in the root of the repo (so the same one as

mage_ai

?). How are people handling this for development? Do they use a single virtual environment for both

mage_ai

and

mage_integrations

? Thanks!

I also wanted to note that, according to the

setup.py

, the lowest compatible version of Python is Python 3.6. I’m pretty sure this is incompatible with at least some of the dependencies in the

requirements.txt

(like

polars

- which looks like it requires at least 3.8; or we should pin the version of

polars

); I ran into installation issues when trying to install the same dependencies with

poetry

. And some of the dependencies in there have known security vulnerabilities and need to be updated (

snyk

told me). I can put in a couple issues around these for the same.

Are there plans to formally switch to the

pyproject.toml

specification for builds of

mage_ai

and

mage_integrations

?

Hello. I started using Mage a couple of months ago. So far its been a fun experience and I am looking to contribute to the documentation.

hey @Xiaoyou Wang i left a couple of PRs in https://github.com/mage-ai/helm-charts would love get some feedback.

hey just added some config for the rediness and liveness probes to helm_chart here. Is there a btter api endpoint to check for readiness? It is reccomended to have a different api endpoint for each probe • livenessProbe -> Checks whether the app is live. The current endpoint

/api/status

does this • readinessProbe -> Checks whether the app is ready. This means, initial startup scripts have been run (e.g. git sync and database setup/migration). I guess for now the

/api/status

will work, but is there a better one to use instead? If not should we add one?

hey @Xiaoyou Wang I found your PR regarding the nonroot docker image, which hasnt been worked on for a while and is marked as paused on the roadmap. I would like to pick it up. Probably in a new PR. Any issues? Also I got some questions/remakrs regarding the implementation: 1. As Mage allows installing additional packages for both python and R, should we move the the packages also to the nonroot? Both are easily done. 2. The order of the Dockerfile statements doesnt seem to be ordered in any way. I would propose ordering them in a way that things, which do not change a lot are processed first, to reduce the image build time 3. The Image has a log of layers, which increases build size, I would propose to reduce them to a fair amount (grouping some stuff into single RUN comments concatenated using

&&

4. Most packages are not pinned, which could accidentally break the image and increase build time, as always the newest version is used, even if there is no need for updating it 5. In generell the Dockerfile doesnt follow the most best practices -> have a look e.g. at hadolint for checking the Dockerfile 6. The nonroot User should not have a fixed UID but rather any UID in the system range 100:65535 -> this can be done by adding

--system

to the

useradd

Would you rather have a single PR for all of it, or should we go one PR for each subtopic, so its easier for you to review and check?

This message was deleted.