When extracting data from a MS SQL Server instance, there's a date column that is not being correctly displayed in the Mage UI (and hence cannot be plotted). After exporting to BigQuery, it's correctly displayed again in the GCP Console. This is how it looks in Mage. Is there a way to display it as a YYYY-MM-DD date as it should be?

I'm getting these errors when deploying to Cloud Run "Default STARTUP TCP probe failed 1 time consecutively for container "test-ppln-1" on port 8080. The instance was not started." and "Ready condition status changed to False for Revision test-ppln-00006-rvt with message: The user-provided container failed to start and listen on the port defined provided by the PORT=8080 environment variable. Logs for this revision might contain more information.". Cloud Run is pulling an image from Artifact registry. I tested the image locally, everything worked fine. Anyone able to help?

is it possible to pull data from a block from a trigger run similar to this?

from mage_ai.data_preparation.variable_manager import get_variable


df = get_variable('iterable_event', 'transform_iterable_base_data', 'output_0')

I have an exporter block that ran successfully but some of the data didn't get exported. I'm wondering if there's something up with those particular rows and want to query the data that was fed into that block.

I have a pipeline with a API call trigger that I deployed to GCP Artifact Registry and that's running on Cloud Run. However, when I launch the web server from Cloud Run, it says that there are no available triggers. Anyone know how to include the trigger when deploying using Docker? Here's my Dockerfile for context.

Hey! I'm new to mage, and I'm trying to push data to Google BigQuery using data exporter and I'm sure my credentials are right, but somehow I keep getting this error, anyone knows why? Thanks. ValueError: ('Could not deserialize key data. The data may be in an incorrect format, it may be encrypted with an unsupported algorithm, or it may be an unsupported key type (e.g. EC curves with explicit parameters).', [_OpenSSLErrorWithText(code=151580774, lib=9, reason=102, reason_text=b'error0908F066PEM routinesget header and databad end line')])

Hello ! Is it possible to share information in memory between the pipes ?

Hello is it possible to runs a pipeline after another finish?

Hey team! I have some containers hosted onto a container registry. Is it possible that I can pull these and run with Mage? Or do I need to extract the code and put within a Mage pipeline?

Hello, I have found a weird issue with Data Integration. I had multiple Data Integrations running in MageAI, which were working perfectly fine, until I restarted my PC. When the MageAI Pod went up, my syncs (some of them, weirdly) started to give this error. Strangely, I never seen this issue before, and my connections are working well, and haven't changed anything. I was trying to upgrade to 0.9.15, but it didn't resolve the issue. It doesn't even want to try running any blocks.

I have an service running in ECS, I installed spark and now im getting the following error

ERROR:mage_ai.api.logging:[2023-08-20T16:43:09.493412][api.views] [2023-08-20 16:43:09.493401] [ERROR] GET /kernels [user: N/A]: {'code': 403, 'message': 'Invalid API key.', 'type': 'invalid_api_key'}

What can I do ?

Hi! Wanted to confirm if setting

variables_retention_period

and

logging_config.retention_period

requires setting up a cron job to run the respective commands, or is Mage set up to do this automatically already? Also, is there a default value for either of those fields? Thanks!

What are the scalability limitation of Mage pipelines locally when deployed to a large cloud instance. Namely, how many pipelines is it optimized for? And what are issues encountered when trying to scale to 10k... 1M... 100M... runs within a day? (assuming that there is sufficient compute to run that many instances of python kernels). I'm about to explore it, but I thought to ask for any insights first.

I have to access mage, which is on AWS EC2 from various system, but I am not able to access as only able to access for one user but giving black screen for another user.

Good Morning all (from CET)! I had a question regarding backing up the mage instance. I want to take a snapshot of the running instance and wanted to know if it is enough to just copy the entire working folder of the mage instance to a different location. Or is there something more nuanced that needs doing? Are secrets / config / pipelines / transformations all preserved?

Hi 😄 ! I stored my secret in Mage and I can access my db but when I go on the pipeline, I face this issue :

Good Afternoon,

This is my first post, I am having issues with a key error. Below are some screenshots of my code and the traceback as well. I have tested this code in a Jupyter Notebook and it successfully adds and prints a new zip_code column to the dataframe. I was wondering if this is a specific issue with Mage or I am doing something wrong previous in the pipeline.

Hello guys: I am exporting data from an S3 to a mongo db using data integration pipeline and getting the following errors: • I have a column that is a number but need to store as an string so I set up the type in the stream as a string but when reach mongo is an INT • i set the destination table name but is still using the stream name to create the table • I establish the unique conflict method to update but get duplicated data in the final table • When the proceeds of saving the data ends, I receive an error, even thou I can see the data was save in my mongoldb.

Do u need help with this?

Hi everyone, Im really new to using mage and trying to get started with some test environments. I am extracting data from a Big Query database and inserting it into a postgres db on my pc. I am trying to set up an incremental load so that only new records are pulled into my postgres db but the pipeline I have set up pulls the entire table every time and duplicates the data. Would someone be able to help or give me some pointers on what I might be doing wrong?

Hello team, I am trying to add Data integration with MSSQL, however, I am getting the following error after entering the connection requirement key and value in Configuration

Error

Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/mage_integrations/sources/mssql/__init__.py", line 1, in <module> from mage_integrations.connections.mssql import ( File "/usr/local/lib/python3.10/site-packages/mage_integrations/connections/mssql/__init__.py", line 2, in <module> import pymssql File "/usr/local/lib/python3.10/site-packages/pymssql/__init__.py", line 3, in <module> from ._pymssql import * File "src/pymssql/_pymssql.pyx", line 1, in init pymssql._pymssql File "src/pymssql/_mssql.pyx", line 99, in genexpr File "src/pymssql/_mssql.pyx", line 99, in genexpr ValueError: invalid literal for int() with base 10: 'dev1194'

Would it be possible if someone can tell me what I have done wrong? The environment is Mage deployed to k8s as pod as v0.9.9. Cheers!

where to set AWS_DEFAULT_REGION for streaming pipeline ?

Hiii im facing this error when going to trigger and run the pipelines. It all works well when I only execute the block from the edit pipeline. Can someone give me some advice on this?

hello, this is my first time using mage, watching online tutorials I see that the code blocks show editable scripts, however, my mage session shows a collapsed code cell that doesn't expand, is there a UI element I am missing here. also, there is a recurring error in the terminal "AttributeError: type object 'object' has no attribute 'spawn'"

Hi , I am getting this exception while executing the dbt model , can anyone help me with this

Hello! Can you tell me why, when exporting data, the exporter accesses a database column that does not exist?

Hi, I am trying to complete this tutorial: https://docs.mage.ai/tutorials/setup-dbt ....to test setting up a dbt pipeline. 1. I can not get past the step 2. That being said, I do not have a local instance running, rather my DevOps spun up a testing instance via Kubernetes. a. Is this related to how the mage instance was set up? I'm guessing rights? 2. I've installed dbt and connected to a test db I want to later use (but I suppose I don't need to do this / am doing it the wrong way?). Anyways, then I manually recreated the folder structure as outlined in the tutorial including fully populated profiles.yaml. a. I am able to complete step 5 but Step 6 is a no go. When I look to add a DBT Model, there is no pop up. b. Altogether, I am not surprised by that behaviour, as no where is a dbt model stored in my repo, but since it's in the tutorial...... i. What am I doing wrong? Related to Issue 1? Sorry if these are some nooby questions; trying to make some progress with testing without constantly pinging my DevOps which will earn me a slap on the fingers :)

Just realized this was already discussed previously: https://mageai.slack.com/archives/C03HTTWFEKE/p1690396104119439 But unresolved or history is lost as no more -niche channel?

Good morning folks, I was wondering what's the fastest way to export data to BigQuery in the Data Integration pipeline. I was using

Run stream in parallel

for all my sample tables and then

disable_update_column_types: false
use_batch_load: false

in the destination tab. It's taking an irrational amount of time (750 rows per minute for a 4 million rows table). Is there a way to make it faster? When I use the batch (default) pipeline it takes like 10 seconds.

hi, How can I assign gpu in a pipeline in mage?