This short post is part of my coding session, where I review the usage of ID and KEY (UUID) fields for my research project. In particular, I will remove the ID fields’ usage and check if the system works as expected. How to remove the GORM ID field? Here is an example of Model transofrmation. […]
In this video session, I am working on CodeKN.com and, in particular, its URL Meta Parser worker. During the coding session, I will set the size for MySQL index fields using Golang GORM models, automate table alter execution for setting collation on the table level, and also debug the application to find errors with page […]
New Coding Session: Add new database tables using Golang and GORM:– Using Go and Gorm save links and image URLs from the website.– Introduce a unique index for a pair of fields (MySQL).– Error handler and logging.
In the attached video I discuss the following topics: Save links and images from the webpage. Mark URL as complete in the pipeline once it is fully parsed. Refactor: extract text compressor to the separate function (similar to decompressor). To-do: define a task for the “webpage data parser” worker. Compress using GZIP We retrieve the […]
I started this video as a review of my previous day – part of the January Coding Challenge. But in the end, I started covering code review topics for my ETL pipeline for fetching and processing URLs. I also cover how I plan my coding tasks, how I use GitHub issues and projects for keeping […]
Golang: PubSub implementation using Go channels. Go PubSub implementation: Data-pipeline Read more about data-pipeline implementation used as an example: Used Resources https://eli.thegreenplace.net/2020/pubsub-using-channels-in-go/ https://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pattern
Last month, I worked on the initial version of the data-pipeline for CodeKN.com’s text-processor. At the current version, it fetches URLs, does basic validation for eligible content, and then passes it for further text-analysis. Each of these rectangles on the image above is a separate step in the pipeline. To connect these steps, I use […]