Waylon WalkerJan 20 Originally published at waylonwalker.com ・3 min read
How small can a minimum kedro pipeline ready to package be? I made one within 4 files that you can pip install. It’s only a total of 35 lines of python, 8 in setup.py
and 27 in mini_kedro_pipeline.py
.
I have everything for this post hosted in this gihub repo, you can fork it, clone it, or just follow along.
pip install git+https://github.com/WaylonWalker/mini-kedro-pipeline
This repo represents the minimal amount of structure to build a kedro pipeline that can be shared across projects. Its installable, and drops right into your hooks.py
or…
Waylon WalkerJan 19 Originally published at waylonwalker.com ・3 min readEditManage
I am making another push in 2021 to get my content out in the world and meeting users where they are. See how I plan to execute.
My content is written in markdown, all markdown. I find that markdown does a really great job at getting out of the way and letting ideas flow onto the page. I am never fussing with fonts and formatting while physically writing posts. …
#vim
Vim’s (neovim in my case) location list can provide some very useful information while developing. Mine gives me information about linting and type checking errors with fairly little config. Generally, it sits nicely at the bottom of the screen and barely affects me. Other times, especially while zoomed way in during a presentation, it just gets in the way.
In python data science/engineering most of our data is in the form of some sort of table, typically a DataFrame from a library like pandas, spark, or dask.
These containers for data contain many convenient methods to manipulate table-like data structures. Sometimes we leverage other data types, namely vanilla
types like lists and dicts, or even numpy data types.
unfamiliar with kedro, check out this post
There are times when our data doesn’t fit nicely into a DataFrame. Lucky for us Kedro has pickle support out of the box. Pickle is a way to store any python object to disk…
What does it take to create an installable python package that can be hosted on pypi?
This post is somewhat inspired by the bottle framework, which is famously created as a single python module. Yes, a whole web framework is written in one file.
.
├── setup.py
└── my_pipeline.py
from setuptools import setupsetup(
name="",
version="0.1.0",
py_modules=["my_pipeline", ],
install_requires=["kedro"],
)
The name of the package can contain any letters, numbers, “_”, or “-”. Even if it’s for internal/personal consumption only I usually check for discrepancy with pypi so that you don’t run into conflicts.
Note that pypi treats…
I wanted a super simple way to cross-link blog posts that require as little effort as possible, yet still looks good in vanilla markdown in GitHub. I have been using a snippet that puts HTML into the markdown. While this works, it’s more manual/difficult for me does not look the best, and does not read well as
The new card should be fully automated to expand with title, description, and
cover image. Bonus if I am able to attach a comment behind it.
If you can call it a card 🤣. This…
Replacing text based on whats in the current search register is a quite handy tool that I use often. I believe I picked this tip up from Nic Janetakis, check out his YouTube channel for some amazing vim tips.
If there is one thing that I Like most about vim it’s the ability to hack on it and make it work well for you.
Vim can often be a bit verbose, but that’s ok because we can hack on it, and make our own shortcuts and keybindings. For instance, finding and replacing text requires using a command at the…
I use my ipython terminal daily. It’s my go to way of running python most of the time. After you use it for a little bit you will probably want to setup a bit of your own configuration.
Activate your virtual environment of choice and pip install it. Any time you are running your project in a virtual environment, you will need to install ipython inside it to access those packages from ipython.
pip install ipython
You are using a virtual environment right? Virtual environments like venv or conda can save you a ton of pain down the road.
When…
https://waylonwalker.com/ I am a self taught python developer, with a background in Mechanical Engineering. During the day I mostly do Data Engineering, with a