Categories
Building CodeKN.com Golang Tutorials Videos

[Video] Go & GORM: Remove ID field and review gorm.Model usage

This short post is part of my coding session, where I review the usage of ID and KEY (UUID) fields for my research project. In particular, I will remove the ID fields’ usage and check if the system works as expected. How to remove the GORM ID field? Here is an example of Model transofrmation. […]

Categories
Building CodeKN.com Coding Session Golang Videos

Golang GORM: Set size for index columns, automate table alter scripts (Coding Session)

In this video session, I am working on CodeKN.com and, in particular, its URL Meta Parser worker. During the coding session, I will set the size for MySQL index fields using Golang GORM models, automate table alter execution for setting collation on the table level, and also debug the application to find errors with page […]

Categories
Building CodeKN.com Coding Session Golang Videos

[Video] Add new database tables using Golang and GORM (MySQL)

New Coding Session: Add new database tables using Golang and GORM:– Using Go and Gorm save links and image URLs from the website.– Introduce a unique index for a pair of fields (MySQL).– Error handler and logging.

Categories
Building CodeKN.com Coding Session Videos

[Video] GZIP compress web page’s content and save in MySQL using GORM (Golang)

In the attached video I discuss the following topics: Save links and images from the webpage. Mark URL as complete in the pipeline once it is fully parsed. Refactor: extract text compressor to the separate function (similar to decompressor). To-do: define a task for the “webpage data parser” worker. Compress using GZIP We retrieve the […]

Categories
Building CodeKN.com Daily Log Videos

[Video] How I plan my code tasks? Code review for data streaming pipeline for URL validation and fetching

I started this video as a review of my previous day – part of the January Coding Challenge. But in the end, I started covering code review topics for my ETL pipeline for fetching and processing URLs. I also cover how I plan my coding tasks, how I use GitHub issues and projects for keeping […]

Categories
Building CodeKN.com Golang Tutorials Videos

[Video] Golang: PubSub implementation using Go channels (Tutorial)

Golang: PubSub implementation using Go channels. Go PubSub implementation: Data-pipeline Read more about data-pipeline implementation used as an example: Used Resources https://eli.thegreenplace.net/2020/pubsub-using-channels-in-go/ https://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pattern

Categories
Building CodeKN.com Data Processing

Work on data-pipeline for fetching & processing URLs

Last month, I worked on the initial version of the data-pipeline for CodeKN.com’s text-processor. At the current version, it fetches URLs, does basic validation for eligible content, and then passes it for further text-analysis. Each of these rectangles on the image above is a separate step in the pipeline. To connect these steps, I use […]