Categories
Building CodeKN.com Data Processing

Work on data-pipeline for fetching & processing URLs

Last month, I worked on the initial version of the data-pipeline for CodeKN.com’s text-processor. At the current version, it fetches URLs, does basic validation for eligible content, and then passes it for further text-analysis. Each of these rectangles on the image above is a separate step in the pipeline. To connect these steps, I use […]