{"users":[{"id":1,"username":"admin","name":"Admin","avatar_template":"https://avatars.discourse-cdn.com/v4/letter/a/3ec8ea/{size}.png","admin":true,"moderator":true,"trust_level":4},{"id":9,"username":"Tarika","name":null,"avatar_template":"https://avatars.discourse-cdn.com/v4/letter/t/8491ac/{size}.png","moderator":true,"trust_level":1},{"id":2,"username":"jayant","name":null,"avatar_template":"https://avatars.discourse-cdn.com/v4/letter/j/ecb155/{size}.png","moderator":true,"trust_level":1},{"id":8,"username":"Ragita","name":null,"avatar_template":"https://avatars.discourse-cdn.com/v4/letter/r/f05b48/{size}.png","moderator":true,"trust_level":1}],"primary_groups":[],"flair_groups":[],"topic_list":{"can_create_topic":false,"filter":"latest","more_topics_url":"/c/data-preparation/10?page=1","per_page":30,"top_tags":[{"id":3,"name":"faq","slug":"faq"},{"id":23,"name":"workflow","slug":"workflow"}],"topics":[{"fancy_title":"About the Data Preparation category","id":16,"title":"About the Data Preparation category","slug":"about-the-data-preparation-category","posts_count":1,"reply_count":0,"highest_post_number":1,"image_url":null,"created_at":"2025-12-08T21:42:09.948Z","last_posted_at":null,"bumped":true,"bumped_at":"2025-12-08T21:42:09.948Z","archetype":"regular","unseen":false,"pinned":true,"unpinned":null,"excerpt":null,"visible":true,"closed":false,"archived":false,"bookmarked":null,"liked":null,"tags":[],"tags_descriptions":{},"views":17,"like_count":0,"has_summary":false,"last_poster_username":"admin","category_id":10,"op_like_count":0,"pinned_globally":false,"featured_link":null,"is_hot":false,"has_accepted_answer":false,"posters":[{"extras":"latest single","description":"Original Poster, Most Recent Poster","user_id":1,"primary_group_id":null,"flair_group_id":null}]},{"fancy_title":"How to Rename Dynamic Year-Based Columns and Pass Column Lists as Parameters in Sparkflows?","id":310,"title":"How to Rename Dynamic Year-Based Columns and Pass Column Lists as Parameters in Sparkflows?","slug":"how-to-rename-dynamic-year-based-columns-and-pass-column-lists-as-parameters-in-sparkflows","posts_count":1,"reply_count":0,"highest_post_number":1,"image_url":"https://canada1.discourse-cdn.com/flex007/uploads/sparkflows/optimized/1X/11c7dea66d64ea0d0f052263d37654a284874dc6_2_1024x455.jpeg","created_at":"2026-03-02T06:41:46.649Z","last_posted_at":"2026-03-02T06:41:46.711Z","bumped":true,"bumped_at":"2026-03-02T06:41:46.711Z","archetype":"regular","unseen":false,"pinned":false,"unpinned":null,"excerpt":"Question\nThe column name changes every year. For example: \nSum of Q1'25, Sum of Q2'25, Sum of Q3'25, Sum of Q4'25, Sum of FY'25 \nFor year 2026, it becomes: \nSum of Q1'26 \nHow do we read and rename the columns to a fixed …","visible":true,"closed":false,"archived":false,"bookmarked":null,"liked":null,"tags":[{"id":23,"name":"workflow","slug":"workflow"}],"tags_descriptions":{},"views":18,"like_count":0,"has_summary":false,"last_poster_username":"admin","category_id":10,"op_like_count":0,"pinned_globally":false,"featured_link":null,"is_hot":false,"has_accepted_answer":false,"posters":[{"extras":"latest single","description":"Original Poster, Most Recent Poster","user_id":1,"primary_group_id":null,"flair_group_id":null}]},{"fancy_title":"How to write conditions in the GE Decision node?","id":166,"title":"How to write conditions in the GE Decision node?","slug":"how-to-write-conditions-in-the-ge-decision-node","posts_count":1,"reply_count":0,"highest_post_number":1,"image_url":null,"created_at":"2025-12-18T11:39:09.942Z","last_posted_at":"2025-12-18T11:39:09.984Z","bumped":true,"bumped_at":"2025-12-18T11:39:09.984Z","archetype":"regular","unseen":false,"pinned":false,"unpinned":null,"excerpt":"To write conditions in the GE Decision node, follow these steps: \nThe GE Decision node requires an input DataFrame from a CSV file created from GE Results. This is typically done using the Create CSV from the GE Results …","visible":true,"closed":false,"archived":false,"bookmarked":null,"liked":null,"tags":[],"tags_descriptions":{},"views":10,"like_count":0,"has_summary":false,"last_poster_username":"Tarika","category_id":10,"op_like_count":0,"pinned_globally":false,"featured_link":null,"is_hot":false,"has_accepted_answer":false,"posters":[{"extras":"latest single","description":"Original Poster, Most Recent Poster","user_id":9,"primary_group_id":null,"flair_group_id":null}]},{"fancy_title":"I want to get the row count at a given stage. How to achieve this in Sparkflows?","id":165,"title":"I want to get the row count at a given stage. How to achieve this in Sparkflows?","slug":"i-want-to-get-the-row-count-at-a-given-stage-how-to-achieve-this-in-sparkflows","posts_count":1,"reply_count":0,"highest_post_number":1,"image_url":null,"created_at":"2025-12-18T11:24:59.269Z","last_posted_at":"2025-12-18T11:24:59.319Z","bumped":true,"bumped_at":"2025-12-18T11:24:59.319Z","archetype":"regular","unseen":false,"pinned":false,"unpinned":null,"excerpt":"In Sparkflows, we can use the Count processor to get row count at any stage. It can be attached to a processor where count is to be derived and it would print the row count. \nTo use the ‘Count’ Processor, do the followin…","visible":true,"closed":false,"archived":false,"bookmarked":null,"liked":null,"tags":[],"tags_descriptions":{},"views":12,"like_count":0,"has_summary":false,"last_poster_username":"Tarika","category_id":10,"op_like_count":0,"pinned_globally":false,"featured_link":null,"is_hot":false,"has_accepted_answer":false,"posters":[{"extras":"latest single","description":"Original Poster, Most Recent Poster","user_id":9,"primary_group_id":null,"flair_group_id":null}]},{"fancy_title":"I have an employee department dataset containing salary information. I want to identify the minimum and maximum salary for each department and location. How to achieve this in Sparkflows?","id":164,"title":"I have an employee department dataset containing salary information. I want to identify the minimum and maximum salary for each department and location. How to achieve this in Sparkflows?","slug":"i-have-an-employee-department-dataset-containing-salary-information-i-want-to-identify-the-minimum-and-maximum-salary-for-each-department-and-location-how-to-achieve-this-in-sparkflows","posts_count":1,"reply_count":0,"highest_post_number":1,"image_url":null,"created_at":"2025-12-18T11:07:36.805Z","last_posted_at":"2025-12-18T11:07:36.853Z","bumped":true,"bumped_at":"2025-12-18T11:07:36.853Z","archetype":"regular","unseen":false,"pinned":false,"unpinned":null,"excerpt":"In Sparkflows, we can use the Multi Windows Analytics processor to compute minimum and maximum values. First it would create a partition by department and location. Then it would compute minimum and maximum salary values …","visible":true,"closed":false,"archived":false,"bookmarked":null,"liked":null,"tags":[],"tags_descriptions":{},"views":9,"like_count":0,"has_summary":false,"last_poster_username":"Tarika","category_id":10,"op_like_count":0,"pinned_globally":false,"featured_link":null,"is_hot":false,"has_accepted_answer":false,"posters":[{"extras":"latest single","description":"Original Poster, Most Recent Poster","user_id":9,"primary_group_id":null,"flair_group_id":null}]},{"fancy_title":"I have an employee department dataset containing salary information. I want to get a salary based ranking within each department and location. How to achieve this in Sparkflows?","id":163,"title":"I have an employee department dataset containing salary information. I want to get a salary based ranking within each department and location. How to achieve this in Sparkflows?","slug":"i-have-an-employee-department-dataset-containing-salary-information-i-want-to-get-a-salary-based-ranking-within-each-department-and-location-how-to-achieve-this-in-sparkflows","posts_count":1,"reply_count":0,"highest_post_number":1,"image_url":null,"created_at":"2025-12-18T11:03:42.112Z","last_posted_at":"2025-12-18T11:03:42.158Z","bumped":true,"bumped_at":"2025-12-18T11:03:42.158Z","archetype":"regular","unseen":false,"pinned":false,"unpinned":null,"excerpt":"In Sparkflows, we can use the Multi Windows Ranking processor to get ranking within a partition. First it would create a partition by department and location. Then it would rank based on salary. \nTo use the ‘Multi Window…","visible":true,"closed":false,"archived":false,"bookmarked":null,"liked":null,"tags":[],"tags_descriptions":{},"views":10,"like_count":0,"has_summary":false,"last_poster_username":"Tarika","category_id":10,"op_like_count":0,"pinned_globally":false,"featured_link":null,"is_hot":false,"has_accepted_answer":false,"posters":[{"extras":"latest single","description":"Original Poster, Most Recent Poster","user_id":9,"primary_group_id":null,"flair_group_id":null}]},{"fancy_title":"In my dataset I have salaries for employees containing various decimal values. I want to round it off to 2 decimal values. How to achieve this in Sparkflows?","id":162,"title":"In my dataset I have salaries for employees containing various decimal values. I want to round it off to 2 decimal values. How to achieve this in Sparkflows?","slug":"in-my-dataset-i-have-salaries-for-employees-containing-various-decimal-values-i-want-to-round-it-off-to-2-decimal-values-how-to-achieve-this-in-sparkflows","posts_count":1,"reply_count":0,"highest_post_number":1,"image_url":null,"created_at":"2025-12-18T10:50:17.506Z","last_posted_at":"2025-12-18T10:50:17.547Z","bumped":true,"bumped_at":"2025-12-18T10:50:17.547Z","archetype":"regular","unseen":false,"pinned":false,"unpinned":null,"excerpt":"In Sparkflows, you can use the Round Value processor to round off values to desired decimal places. \nTo use the ‘Round Value’ Processor, do the following: \n\n\nSelect columns to be rounded off in ‘Input Column’. It can be …","visible":true,"closed":false,"archived":false,"bookmarked":null,"liked":null,"tags":[],"tags_descriptions":{},"views":18,"like_count":0,"has_summary":false,"last_poster_username":"Tarika","category_id":10,"op_like_count":0,"pinned_globally":false,"featured_link":null,"is_hot":false,"has_accepted_answer":false,"posters":[{"extras":"latest single","description":"Original Poster, Most Recent Poster","user_id":9,"primary_group_id":null,"flair_group_id":null}]},{"fancy_title":"How to split the string value in a column into multiple columns in Sparkflows?","id":161,"title":"How to split the string value in a column into multiple columns in Sparkflows?","slug":"how-to-split-the-string-value-in-a-column-into-multiple-columns-in-sparkflows","posts_count":1,"reply_count":0,"highest_post_number":1,"image_url":null,"created_at":"2025-12-18T10:35:31.524Z","last_posted_at":"2025-12-18T10:35:31.564Z","bumped":true,"bumped_at":"2025-12-18T10:35:31.564Z","archetype":"regular","unseen":false,"pinned":false,"unpinned":null,"excerpt":"Sparkflows provides the node called Field Splitter to create multiple columns from a single column provided by a separator. \nTo know more, refer here: Parse Functions — Sparkflows 3.0 documentation","visible":true,"closed":false,"archived":false,"bookmarked":null,"liked":null,"tags":[],"tags_descriptions":{},"views":8,"like_count":0,"has_summary":false,"last_poster_username":"Tarika","category_id":10,"op_like_count":0,"pinned_globally":false,"featured_link":null,"is_hot":false,"has_accepted_answer":false,"posters":[{"extras":"latest single","description":"Original Poster, Most Recent Poster","user_id":9,"primary_group_id":null,"flair_group_id":null}]},{"fancy_title":"How to split the input data into two outputs depending on the condition?","id":160,"title":"How to split the input data into two outputs depending on the condition?","slug":"how-to-split-the-input-data-into-two-outputs-depending-on-the-condition","posts_count":1,"reply_count":0,"highest_post_number":1,"image_url":null,"created_at":"2025-12-18T10:32:22.689Z","last_posted_at":"2025-12-18T10:32:22.747Z","bumped":true,"bumped_at":"2025-12-18T10:32:22.747Z","archetype":"regular","unseen":false,"pinned":false,"unpinned":null,"excerpt":"Sparkflows provide the node called Split By Expression to split the input DataFrame into two DataFrames. \nTo know more, read this documentation: Split Dataset By Expression — Sparkflows 3.0 documentation","visible":true,"closed":false,"archived":false,"bookmarked":null,"liked":null,"tags":[],"tags_descriptions":{},"views":4,"like_count":0,"has_summary":false,"last_poster_username":"Tarika","category_id":10,"op_like_count":0,"pinned_globally":false,"featured_link":null,"is_hot":false,"has_accepted_answer":false,"posters":[{"extras":"latest single","description":"Original Poster, Most Recent Poster","user_id":9,"primary_group_id":null,"flair_group_id":null}]},{"fancy_title":"Given a retail dataset, the goal is to validate the schema, address fields, and overall structure to ensure data accuracy and quality for reliable analysis and decision-making. How can this be achieved?","id":159,"title":"Given a retail dataset, the goal is to validate the schema, address fields, and overall structure to ensure data accuracy and quality for reliable analysis and decision-making. How can this be achieved?","slug":"given-a-retail-dataset-the-goal-is-to-validate-the-schema-address-fields-and-overall-structure-to-ensure-data-accuracy-and-quality-for-reliable-analysis-and-decision-making-how-can-this-be-achieved","posts_count":1,"reply_count":0,"highest_post_number":1,"image_url":null,"created_at":"2025-12-18T10:23:07.631Z","last_posted_at":"2025-12-18T10:23:07.671Z","bumped":true,"bumped_at":"2025-12-18T10:23:07.671Z","archetype":"regular","unseen":false,"pinned":false,"unpinned":null,"excerpt":"Sparkflows provides a diverse range of nodes that cater to the above-mentioned requirements as these nodes assist in ensuring data quality and integrity. Some of them are listed below : \n\n\nNode Schema Validation \n\n\nValid…","visible":true,"closed":false,"archived":false,"bookmarked":null,"liked":null,"tags":[],"tags_descriptions":{},"views":11,"like_count":0,"has_summary":false,"last_poster_username":"Tarika","category_id":10,"op_like_count":0,"pinned_globally":false,"featured_link":null,"is_hot":false,"has_accepted_answer":false,"posters":[{"extras":"latest single","description":"Original Poster, Most Recent Poster","user_id":9,"primary_group_id":null,"flair_group_id":null}]},{"fancy_title":"How can I explore data with the help of Sparkflows?","id":158,"title":"How can I explore data with the help of Sparkflows?","slug":"how-can-i-explore-data-with-the-help-of-sparkflows","posts_count":1,"reply_count":0,"highest_post_number":1,"image_url":null,"created_at":"2025-12-18T10:01:51.725Z","last_posted_at":"2025-12-18T10:01:51.766Z","bumped":true,"bumped_at":"2025-12-18T10:01:51.766Z","archetype":"regular","unseen":false,"pinned":false,"unpinned":null,"excerpt":"Sparkflows offers a range of nodes for data profiling and exploratory analysis, allowing users to examine data profiles and perform comprehensive exploration tasks. Those are listed below: \n\nSummary Statistics\nColumns Ca…","visible":true,"closed":false,"archived":false,"bookmarked":null,"liked":null,"tags":[],"tags_descriptions":{},"views":5,"like_count":0,"has_summary":false,"last_poster_username":"Tarika","category_id":10,"op_like_count":0,"pinned_globally":false,"featured_link":null,"is_hot":false,"has_accepted_answer":false,"posters":[{"extras":"latest single","description":"Original Poster, Most Recent Poster","user_id":9,"primary_group_id":null,"flair_group_id":null}]},{"fancy_title":"I possess EHR data which contains nested JSON format. I would like to extract all the fields and convert them into a CSV file. How can I achieve this in Sparkflows?","id":157,"title":"I possess EHR data which contains nested JSON format. I would like to extract all the fields and convert them into a CSV file. How can I achieve this in Sparkflows?","slug":"i-possess-ehr-data-which-contains-nested-json-format-i-would-like-to-extract-all-the-fields-and-convert-them-into-a-csv-file-how-can-i-achieve-this-in-sparkflows","posts_count":1,"reply_count":0,"highest_post_number":1,"image_url":null,"created_at":"2025-12-18T09:50:54.792Z","last_posted_at":"2025-12-18T09:50:54.831Z","bumped":true,"bumped_at":"2025-12-18T09:50:54.831Z","archetype":"regular","unseen":false,"pinned":false,"unpinned":null,"excerpt":"You can use the Flatten and Explode nodes. The “Flatten” node allows to convert complex nested structures into a more straightforward columnar format. On the other hand, the “Explode” node is useful for breaking down arr…","visible":true,"closed":false,"archived":false,"bookmarked":null,"liked":null,"tags":[],"tags_descriptions":{},"views":10,"like_count":0,"has_summary":false,"last_poster_username":"Tarika","category_id":10,"op_like_count":0,"pinned_globally":false,"featured_link":null,"is_hot":false,"has_accepted_answer":false,"posters":[{"extras":"latest single","description":"Original Poster, Most Recent Poster","user_id":9,"primary_group_id":null,"flair_group_id":null}]},{"fancy_title":"I want to use only a smaller set of data for my analysis. How to achieve this in Sparkflows?","id":156,"title":"I want to use only a smaller set of data for my analysis. How to achieve this in Sparkflows?","slug":"i-want-to-use-only-a-smaller-set-of-data-for-my-analysis-how-to-achieve-this-in-sparkflows","posts_count":1,"reply_count":0,"highest_post_number":1,"image_url":null,"created_at":"2025-12-18T09:45:30.153Z","last_posted_at":"2025-12-18T09:45:30.207Z","bumped":true,"bumped_at":"2025-12-18T09:45:30.207Z","archetype":"regular","unseen":false,"pinned":false,"unpinned":null,"excerpt":"In Sparkflows, we can use the Sample processor to extract a sample of incoming datasets. The number of rows in the sample would be a percentage of the incoming dataset. Sample can be used for ML Training or Analysis purp…","visible":true,"closed":false,"archived":false,"bookmarked":null,"liked":null,"tags":[],"tags_descriptions":{},"views":4,"like_count":0,"has_summary":false,"last_poster_username":"Tarika","category_id":10,"op_like_count":0,"pinned_globally":false,"featured_link":null,"is_hot":false,"has_accepted_answer":false,"posters":[{"extras":"latest single","description":"Original Poster, Most Recent Poster","user_id":9,"primary_group_id":null,"flair_group_id":null}]},{"fancy_title":"I want to sort incoming dataset. How to achieve this in Sparkflows?","id":155,"title":"I want to sort incoming dataset. How to achieve this in Sparkflows?","slug":"i-want-to-sort-incoming-dataset-how-to-achieve-this-in-sparkflows","posts_count":1,"reply_count":0,"highest_post_number":1,"image_url":null,"created_at":"2025-12-18T09:32:35.299Z","last_posted_at":"2025-12-18T09:32:35.347Z","bumped":true,"bumped_at":"2025-12-18T09:32:35.347Z","archetype":"regular","unseen":false,"pinned":false,"unpinned":null,"excerpt":"In Sparkflows, we can use the Sort By processor to sort incoming dataset. Dataset can be sorted based on one or multiple columns. \nTo use the ‘Sort By’ Processor: \n\n\nSelect a column and sorting order to sort the incoming…","visible":true,"closed":false,"archived":false,"bookmarked":null,"liked":null,"tags":[],"tags_descriptions":{},"views":7,"like_count":0,"has_summary":false,"last_poster_username":"Tarika","category_id":10,"op_like_count":0,"pinned_globally":false,"featured_link":null,"is_hot":false,"has_accepted_answer":false,"posters":[{"extras":"latest single","description":"Original Poster, Most Recent Poster","user_id":9,"primary_group_id":null,"flair_group_id":null}]},{"fancy_title":"How can I transpose a dataset in Sparkflows?","id":145,"title":"How can I transpose a dataset in Sparkflows?","slug":"how-can-i-transpose-a-dataset-in-sparkflows","posts_count":1,"reply_count":0,"highest_post_number":1,"image_url":null,"created_at":"2025-12-17T11:46:54.031Z","last_posted_at":"2025-12-17T11:46:54.075Z","bumped":true,"bumped_at":"2025-12-17T11:46:54.075Z","archetype":"regular","unseen":false,"pinned":false,"unpinned":null,"excerpt":"In Sparkflows, we can use the Transpose processor to transpose a dataset. \nTo use the ‘Transpose’ Processor: \n\n\nSelect a column to be used to transpose incoming dataset in ‘Transpose By Column Name’. \n\n\nOutput would be a…","visible":true,"closed":false,"archived":false,"bookmarked":null,"liked":null,"tags":[],"tags_descriptions":{},"views":16,"like_count":0,"has_summary":false,"last_poster_username":"Tarika","category_id":10,"op_like_count":0,"pinned_globally":false,"featured_link":null,"is_hot":false,"has_accepted_answer":false,"posters":[{"extras":"latest single","description":"Original Poster, Most Recent Poster","user_id":9,"primary_group_id":null,"flair_group_id":null}]},{"fancy_title":"I want to sort incoming columns in a particular order","id":144,"title":"I want to sort incoming columns in a particular order","slug":"i-want-to-sort-incoming-columns-in-a-particular-order","posts_count":1,"reply_count":0,"highest_post_number":1,"image_url":null,"created_at":"2025-12-17T11:40:43.443Z","last_posted_at":"2025-12-17T11:40:43.485Z","bumped":true,"bumped_at":"2025-12-17T11:40:43.485Z","archetype":"regular","unseen":false,"pinned":false,"unpinned":null,"excerpt":"In Sparkflows, you can use the Sort Columns processor to sort incoming columns in any order. \nYou can use the ‘Sort Columns’ processor as below: \n\nColumns can be sorted in Ascending or Descending order of column names.\nC…","visible":true,"closed":false,"archived":false,"bookmarked":null,"liked":null,"tags":[],"tags_descriptions":{},"views":15,"like_count":0,"has_summary":false,"last_poster_username":"Tarika","category_id":10,"op_like_count":0,"pinned_globally":false,"featured_link":null,"is_hot":false,"has_accepted_answer":false,"posters":[{"extras":"latest single","description":"Original Poster, Most Recent Poster","user_id":9,"primary_group_id":null,"flair_group_id":null}]},{"fancy_title":"I want to perform state-wise analysis on a dataset related to population","id":143,"title":"I want to perform state-wise analysis on a dataset related to population","slug":"i-want-to-perform-state-wise-analysis-on-a-dataset-related-to-population","posts_count":1,"reply_count":0,"highest_post_number":1,"image_url":null,"created_at":"2025-12-17T11:33:12.008Z","last_posted_at":"2025-12-17T11:33:12.048Z","bumped":true,"bumped_at":"2025-12-17T11:33:12.048Z","archetype":"regular","unseen":false,"pinned":false,"unpinned":null,"excerpt":"In Sparkflows, we can use the Windows Aggregation processor to perform state wise data analysis. Data can be partitioned by State and various computations can be performed such as avg, max, min, range, std dev, and so on…","visible":true,"closed":false,"archived":false,"bookmarked":null,"liked":null,"tags":[],"tags_descriptions":{},"views":16,"like_count":0,"has_summary":false,"last_poster_username":"Tarika","category_id":10,"op_like_count":0,"pinned_globally":false,"featured_link":null,"is_hot":false,"has_accepted_answer":false,"posters":[{"extras":"latest single","description":"Original Poster, Most Recent Poster","user_id":9,"primary_group_id":null,"flair_group_id":null}]},{"fancy_title":"How can I perform Windows Analytics on a dataset in Sparkflows?","id":142,"title":"How can I perform Windows Analytics on a dataset in Sparkflows?","slug":"how-can-i-perform-windows-analytics-on-a-dataset-in-sparkflows","posts_count":1,"reply_count":0,"highest_post_number":1,"image_url":null,"created_at":"2025-12-17T11:28:50.917Z","last_posted_at":"2025-12-17T11:28:50.968Z","bumped":true,"bumped_at":"2025-12-17T11:28:50.968Z","archetype":"regular","unseen":false,"pinned":false,"unpinned":null,"excerpt":"In Sparkflows, we can use the ‘Windows Analytics’ processor to perform Windows Analytics on a dataset. It facilitates partitioning the dataset based on a selected column and applies windows functions such as first_value,…","visible":true,"closed":false,"archived":false,"bookmarked":null,"liked":null,"tags":[],"tags_descriptions":{},"views":15,"like_count":0,"has_summary":false,"last_poster_username":"Tarika","category_id":10,"op_like_count":0,"pinned_globally":false,"featured_link":null,"is_hot":false,"has_accepted_answer":false,"posters":[{"extras":"latest single","description":"Original Poster, Most Recent Poster","user_id":9,"primary_group_id":null,"flair_group_id":null}]},{"fancy_title":"How do I split my data into unique and duplicate records in Sparkflows?","id":141,"title":"How do I split my data into unique and duplicate records in Sparkflows?","slug":"how-do-i-split-my-data-into-unique-and-duplicate-records-in-sparkflows","posts_count":1,"reply_count":0,"highest_post_number":1,"image_url":"https://canada1.discourse-cdn.com/flex007/uploads/sparkflows/original/1X/0830e072c8512a8e24c850f5ed0188411054e779.png","created_at":"2025-12-17T10:52:50.584Z","last_posted_at":"2025-12-17T10:52:50.627Z","bumped":true,"bumped_at":"2025-12-17T10:52:50.627Z","archetype":"regular","unseen":false,"pinned":false,"unpinned":null,"excerpt":"In Sparkflows, there is a Find Duplicate node that can perform the exact operation you’re looking for. You can specify the column(s) based on which you want to determine uniqueness. Simply add this node to your input dat…","visible":true,"closed":false,"archived":false,"bookmarked":null,"liked":null,"tags":[],"tags_descriptions":{},"views":4,"like_count":0,"has_summary":false,"last_poster_username":"Tarika","category_id":10,"op_like_count":0,"pinned_globally":false,"featured_link":null,"is_hot":false,"has_accepted_answer":false,"posters":[{"extras":"latest single","description":"Original Poster, Most Recent Poster","user_id":9,"primary_group_id":null,"flair_group_id":null}]},{"fancy_title":"Is there a way to replace a specific value in a column using Sparkflows?","id":140,"title":"Is there a way to replace a specific value in a column using Sparkflows?","slug":"is-there-a-way-to-replace-a-specific-value-in-a-column-using-sparkflows","posts_count":1,"reply_count":0,"highest_post_number":1,"image_url":"https://canada1.discourse-cdn.com/flex007/uploads/sparkflows/original/1X/28ba4de3adc3f61764a8973099fcd94ba307807f.png","created_at":"2025-12-17T09:42:34.160Z","last_posted_at":"2025-12-17T09:42:34.210Z","bumped":true,"bumped_at":"2025-12-17T09:42:34.210Z","archetype":"regular","unseen":false,"pinned":false,"unpinned":null,"excerpt":"To replace a specific value in a column using Sparkflows, you can utilize the Impute Advanced node, which allows you to configure value replacements or imputations. \nTo replace a specific value with a constant, you simpl…","visible":true,"closed":false,"archived":false,"bookmarked":null,"liked":null,"tags":[],"tags_descriptions":{},"views":14,"like_count":0,"has_summary":false,"last_poster_username":"Tarika","category_id":10,"op_like_count":0,"pinned_globally":false,"featured_link":null,"is_hot":false,"has_accepted_answer":false,"posters":[{"extras":"latest single","description":"Original Poster, Most Recent Poster","user_id":9,"primary_group_id":null,"flair_group_id":null}]},{"fancy_title":"How can I filter good and bad records after performing data quality checks using Great Expectation nodes?","id":139,"title":"How can I filter good and bad records after performing data quality checks using Great Expectation nodes?","slug":"how-can-i-filter-good-and-bad-records-after-performing-data-quality-checks-using-great-expectation-nodes","posts_count":1,"reply_count":0,"highest_post_number":1,"image_url":"https://canada1.discourse-cdn.com/flex007/uploads/sparkflows/original/1X/b66925afd8957881260525b32f20ecf3288c8867.png","created_at":"2025-12-17T08:36:29.408Z","last_posted_at":"2025-12-17T08:36:29.450Z","bumped":true,"bumped_at":"2025-12-17T08:36:29.450Z","archetype":"regular","unseen":false,"pinned":false,"unpinned":null,"excerpt":"To separate good and bad records, you can utilize the "Split Into Good Bad Records’’ node. After adding this node to your workflow following any Great Expectation node, ensure that the input DataFrame is the original Dat…","visible":true,"closed":false,"archived":false,"bookmarked":null,"liked":null,"tags":[],"tags_descriptions":{},"views":20,"like_count":0,"has_summary":false,"last_poster_username":"Tarika","category_id":10,"op_like_count":0,"pinned_globally":false,"featured_link":null,"is_hot":false,"has_accepted_answer":false,"posters":[{"extras":"latest single","description":"Original Poster, Most Recent Poster","user_id":9,"primary_group_id":null,"flair_group_id":null}]},{"fancy_title":"How to Normalize data using Sparkflows?","id":130,"title":"How to Normalize data using Sparkflows?","slug":"how-to-normalize-data-using-sparkflows","posts_count":1,"reply_count":0,"highest_post_number":1,"image_url":"https://canada1.discourse-cdn.com/flex007/uploads/sparkflows/optimized/1X/c561996f9767f4a91a754e6256c1923dc36503c2_2_1024x498.png","created_at":"2025-12-16T13:05:47.196Z","last_posted_at":"2025-12-16T13:05:47.241Z","bumped":true,"bumped_at":"2025-12-16T13:05:47.241Z","archetype":"regular","unseen":false,"pinned":false,"unpinned":null,"excerpt":"In Sparkflows, you can use the “Normalizer” node to normalize the data. Let us try to understand the process with the help of a simple workflow. \nBelow is the image of a simple workflow using the Normalizer node. \n  \nBef…","visible":true,"closed":false,"archived":false,"bookmarked":null,"liked":null,"tags":[],"tags_descriptions":{},"views":3,"like_count":0,"has_summary":false,"last_poster_username":"Tarika","category_id":10,"op_like_count":0,"pinned_globally":false,"featured_link":null,"is_hot":false,"has_accepted_answer":false,"posters":[{"extras":"latest single","description":"Original Poster, Most Recent Poster","user_id":9,"primary_group_id":null,"flair_group_id":null}]},{"fancy_title":"I want to get word count from the reviews before they are fed to the database. How can I achieve this in sparkflows?","id":129,"title":"I want to get word count from the reviews before they are fed to the database. How can I achieve this in sparkflows?","slug":"i-want-to-get-word-count-from-the-reviews-before-they-are-fed-to-the-database-how-can-i-achieve-this-in-sparkflows","posts_count":1,"reply_count":0,"highest_post_number":1,"image_url":null,"created_at":"2025-12-16T12:44:36.199Z","last_posted_at":"2025-12-16T12:44:36.246Z","bumped":true,"bumped_at":"2025-12-16T12:44:36.246Z","archetype":"regular","unseen":false,"pinned":false,"unpinned":null,"excerpt":"In Sparkflows, we can use the ‘Word Count’ processor to get word count of selected columns. \nTo use the ‘Word Count’ Processor: \n\nSelect a set of columns for which word count is to be computed in the ‘Input Columns’ fiel…","visible":true,"closed":false,"archived":false,"bookmarked":null,"liked":null,"tags":[],"tags_descriptions":{},"views":23,"like_count":0,"has_summary":false,"last_poster_username":"Tarika","category_id":10,"op_like_count":0,"pinned_globally":false,"featured_link":null,"is_hot":false,"has_accepted_answer":false,"posters":[{"extras":"latest single","description":"Original Poster, Most Recent Poster","user_id":9,"primary_group_id":null,"flair_group_id":null}]},{"fancy_title":"I want to analyze log data to get meaningful insight. How can I achieve this in sparkflows?","id":128,"title":"I want to analyze log data to get meaningful insight. How can I achieve this in sparkflows?","slug":"i-want-to-analyze-log-data-to-get-meaningful-insight-how-can-i-achieve-this-in-sparkflows","posts_count":1,"reply_count":0,"highest_post_number":1,"image_url":null,"created_at":"2025-12-16T12:38:50.391Z","last_posted_at":"2025-12-16T12:38:50.431Z","bumped":true,"bumped_at":"2025-12-16T12:38:50.431Z","archetype":"regular","unseen":false,"pinned":false,"unpinned":null,"excerpt":"In Sparkflows, we can use the “Apache Logs” processor to read and process log data. It reads a log file and loads it as a DataFrame. Thereafter DataFrame can be used for further analysis. \nTo use the “Apache Logs” Proces…","visible":true,"closed":false,"archived":false,"bookmarked":null,"liked":null,"tags":[],"tags_descriptions":{},"views":11,"like_count":0,"has_summary":false,"last_poster_username":"Tarika","category_id":10,"op_like_count":0,"pinned_globally":false,"featured_link":null,"is_hot":false,"has_accepted_answer":false,"posters":[{"extras":"latest single","description":"Original Poster, Most Recent Poster","user_id":9,"primary_group_id":null,"flair_group_id":null}]},{"fancy_title":"I have one year’s worth of sales data, and I’d like to calculate the cumulative sales total, enabling me to analyze the total sales made up to a specific date","id":127,"title":"I have one year's worth of sales data, and I'd like to calculate the cumulative sales total, enabling me to analyze the total sales made up to a specific date","slug":"i-have-one-years-worth-of-sales-data-and-id-like-to-calculate-the-cumulative-sales-total-enabling-me-to-analyze-the-total-sales-made-up-to-a-specific-date","posts_count":1,"reply_count":0,"highest_post_number":1,"image_url":"https://canada1.discourse-cdn.com/flex007/uploads/sparkflows/optimized/1X/b23300257f30a3324a86dbc81b6ecf4003680dd3_2_1024x449.png","created_at":"2025-12-16T12:31:37.435Z","last_posted_at":"2025-12-16T12:31:37.479Z","bumped":true,"bumped_at":"2025-12-16T12:31:37.479Z","archetype":"regular","unseen":false,"pinned":false,"unpinned":null,"excerpt":"You can fulfill the aforementioned requirement by utilizing a window aggregate function through Window Aggregation Node, enabling you to compute the running total of sales. Moreover, you can also calculate the cumulative…","visible":true,"closed":false,"archived":false,"bookmarked":null,"liked":null,"tags":[],"tags_descriptions":{},"views":10,"like_count":0,"has_summary":false,"last_poster_username":"Tarika","category_id":10,"op_like_count":0,"pinned_globally":false,"featured_link":null,"is_hot":false,"has_accepted_answer":false,"posters":[{"extras":"latest single","description":"Original Poster, Most Recent Poster","user_id":9,"primary_group_id":null,"flair_group_id":null}]},{"fancy_title":"How does Apache Spark’s distributed execution affect the number of output files, and what method ensures saving the result in only one file?","id":89,"title":"How does Apache Spark’s distributed execution affect the number of output files, and what method ensures saving the result in only one file?","slug":"how-does-apache-spark-s-distributed-execution-affect-the-number-of-output-files-and-what-method-ensures-saving-the-result-in-only-one-file","posts_count":1,"reply_count":0,"highest_post_number":1,"image_url":null,"created_at":"2025-12-12T14:13:49.995Z","last_posted_at":"2025-12-12T14:13:50.036Z","bumped":true,"bumped_at":"2025-12-12T14:13:50.036Z","archetype":"regular","unseen":false,"pinned":false,"unpinned":null,"excerpt":"Apache Spark runs distributed. As a result the data is partitioned across multiple process/machines. \nWhen any of the Save nodes is used to write the output to files, the number of files created is dependent on the numbe…","visible":true,"closed":false,"archived":false,"bookmarked":null,"liked":null,"tags":[{"id":3,"name":"faq","slug":"faq"}],"tags_descriptions":{},"views":12,"like_count":0,"has_summary":false,"last_poster_username":"Tarika","category_id":10,"op_like_count":0,"pinned_globally":false,"featured_link":null,"is_hot":false,"has_accepted_answer":false,"posters":[{"extras":"latest single","description":"Original Poster, Most Recent Poster","user_id":9,"primary_group_id":null,"flair_group_id":null}]},{"fancy_title":"Regex used to add _ in front number column name","id":80,"title":"Regex used to add _ in front number column name","slug":"regex-used-to-add-in-front-number-column-name","posts_count":1,"reply_count":0,"highest_post_number":1,"image_url":"https://canada1.discourse-cdn.com/flex007/uploads/sparkflows/optimized/1X/39d4002f890d91947b4ac7ede39c507632f7b852_2_1024x419.jpeg","created_at":"2025-12-12T09:32:01.371Z","last_posted_at":"2025-12-12T09:32:01.413Z","bumped":true,"bumped_at":"2025-12-12T09:32:01.413Z","archetype":"regular","unseen":false,"pinned":false,"unpinned":null,"excerpt":"BulkColumnRename node can be used to add _ in front number column name.","visible":true,"closed":false,"archived":false,"bookmarked":null,"liked":null,"tags":[],"tags_descriptions":{},"views":14,"like_count":0,"has_summary":false,"last_poster_username":"jayant","category_id":10,"op_like_count":0,"pinned_globally":false,"featured_link":null,"is_hot":false,"has_accepted_answer":false,"posters":[{"extras":"latest single","description":"Original Poster, Most Recent Poster","user_id":2,"primary_group_id":null,"flair_group_id":null}]},{"fancy_title":"How do I aggregate columns in the workflow designer?","id":75,"title":"How do I aggregate columns in the workflow designer?","slug":"how-do-i-aggregate-columns-in-the-workflow-designer","posts_count":1,"reply_count":0,"highest_post_number":1,"image_url":null,"created_at":"2025-12-11T19:38:19.747Z","last_posted_at":"2025-12-11T19:38:19.802Z","bumped":true,"bumped_at":"2025-12-11T19:38:19.802Z","archetype":"regular","unseen":false,"pinned":false,"unpinned":null,"excerpt":"Columns can be aggregated using the Aggregate node. You can provide an expression for the aggregation. \nYou can also name the aggregated column in the process.","visible":true,"closed":false,"archived":false,"bookmarked":null,"liked":null,"tags":[{"id":3,"name":"faq","slug":"faq"}],"tags_descriptions":{},"views":12,"like_count":0,"has_summary":false,"last_poster_username":"jayant","category_id":10,"op_like_count":0,"pinned_globally":false,"featured_link":null,"is_hot":false,"has_accepted_answer":false,"posters":[{"extras":"latest single","description":"Original Poster, Most Recent Poster","user_id":2,"primary_group_id":null,"flair_group_id":null}]},{"fancy_title":"I would like to extract currency value from this column. How can I achieve this in Sparkflows? I have a dataset containing an amount column in the format ccy + value (USD 1000000.00)","id":68,"title":"I would like to extract currency value from this column. How can I achieve this in Sparkflows? I have a dataset containing an amount column in the format ccy + value (USD 1000000.00)","slug":"i-would-like-to-extract-currency-value-from-this-column-how-can-i-achieve-this-in-sparkflows-i-have-a-dataset-containing-an-amount-column-in-the-format-ccy-value-usd-1000000-00","posts_count":1,"reply_count":0,"highest_post_number":1,"image_url":null,"created_at":"2025-12-11T09:00:09.770Z","last_posted_at":"2025-12-11T09:00:09.819Z","bumped":true,"bumped_at":"2025-12-11T09:00:09.819Z","archetype":"regular","unseen":false,"pinned":false,"unpinned":null,"excerpt":"In Sparkflows, we can use the “Multi Regex Extractor” processor to achieve this. Processor needs to be configured as below and it would extract the currency part. \nTo use the “Multi Regex Extractor” Processor: \n\n\nSelect …","visible":true,"closed":false,"archived":false,"bookmarked":null,"liked":null,"tags":[],"tags_descriptions":{},"views":15,"like_count":0,"has_summary":false,"last_poster_username":"Ragita","category_id":10,"op_like_count":0,"pinned_globally":false,"featured_link":null,"is_hot":false,"has_accepted_answer":false,"posters":[{"extras":"latest single","description":"Original Poster, Most Recent Poster","user_id":8,"primary_group_id":null,"flair_group_id":null}]},{"fancy_title":"I have a dataset having sales information of multiple stores from various locations. How can I rank stores by their sales values within a locality using sparkflows?","id":41,"title":"I have a dataset having sales information of multiple stores from various locations. How can I rank stores by their sales values within a locality using sparkflows?","slug":"i-have-a-dataset-having-sales-information-of-multiple-stores-from-various-locations-how-can-i-rank-stores-by-their-sales-values-within-a-locality-using-sparkflows","posts_count":1,"reply_count":0,"highest_post_number":1,"image_url":null,"created_at":"2025-12-10T06:20:00.721Z","last_posted_at":"2025-12-10T06:20:00.760Z","bumped":true,"bumped_at":"2025-12-10T06:20:00.760Z","archetype":"regular","unseen":false,"pinned":false,"unpinned":null,"excerpt":"In Sparkflows, we can use the ‘Windows Ranking’ processor to rank stores based on their sales value within a location. Dataset can be partitioned by location and sorted by sales values. Using this configuration we can ge…","visible":true,"closed":false,"archived":false,"bookmarked":null,"liked":null,"tags":[],"tags_descriptions":{},"views":15,"like_count":0,"has_summary":false,"last_poster_username":"Ragita","category_id":10,"op_like_count":0,"pinned_globally":false,"featured_link":null,"is_hot":false,"has_accepted_answer":false,"posters":[{"extras":"latest single","description":"Original Poster, Most Recent Poster","user_id":8,"primary_group_id":null,"flair_group_id":null}]}]}}