#blog

Waylon WalkerJan 19 Originally published at waylonwalker.com ・3 min readEditManage

I am making another push in 2021 to get my content out in the world and meeting users where they are. See how I plan to execute.

Platforms

• waylonwalker.com
• Twitter
• DEV
• hashnode
• Medium
• LinkedIn
• Anchor

Markdown

My content is written in markdown, all markdown. I find that markdown does a really great job at getting out of the way and letting ideas flow onto the page. I am never fussing with fonts and formatting while physically writing posts. …

#bash

Recently I automated starting new posts with a python script. Today I want to work on the next part that is editing those posts quickly.

#vim

Vim’s (neovim in my case) location list can provide some very useful information while developing. Mine gives me information about linting and type checking errors with fairly little config. Generally, it sits nicely at the bottom of the screen and barely affects me. Other times, especially while zoomed way in during a presentation, it just gets in the way.

In python data science/engineering most of our data is in the form of some sort of table, typically a DataFrame from a library like pandas, spark, or dask.

DataFrames are the heart of most pipelines

These containers for data contain many convenient methods to manipulate table-like data structures. Sometimes we leverage other data types, namely vanilla
types like lists and dicts, or even numpy data types.

🤷‍♀️ What is Kedro (The Parts)

unfamiliar with kedro, check out this post

Sometimes datasets are not tables

There are times when our data doesn’t fit nicely into a DataFrame. Lucky for us Kedro has pickle support out of the box. Pickle is a way to store any python
object to disk. Beware…

What does it take to create an installable python package that can be hosted on pypi?

What is the minimal python package

• setup.py
• my_module.py

This post is somewhat inspired by the bottle framework, which is famously created as a single python module. Yes, a whole web framework is written in one file.

Directory structure

.
├── setup.py
└── my_pipeline.py

setup.py

from setuptools import setupsetup(
    name="",
    version="0.1.0",
    py_modules=["my_pipeline", ],
    install_requires=["kedro"],
)

name

The name of the package can contain any letters, numbers, “_”, or “-”. Even if it’s for internal/personal consumption only I usually check for discrepancy with pypi so that you don’t run into conflicts.

Note that pypi treats…

I wanted a super simple way to cross-link blog posts that require as little effort as possible, yet still looks good in vanilla markdown in GitHub. I have been using a snippet that puts HTML into the markdown. While this works, it’s more manual/difficult for me does not look the best, and does not read well as

Goals for new card

The new card should be fully automated to expand with title, description, and
cover image. Bonus if I am able to attach a comment behind it.

• fully automated
• card expansion
• Title
• description
• cover image

Old Card

If you can call it a card 🤣. This…

Replacing text based on whats in the current search register is a quite handy tool that I use often. I believe I picked this tip up from Nic Janetakis, check out his YouTube channel for some amazing vim tips.

If there is one thing that I Like most about vim it’s the ability to hack on it and make it work well for you.

Replacing text in vim

Vim can often be a bit verbose, but that’s ok because we can hack on it, and make our own shortcuts and keybindings. For instance, finding and replacing text requires using a command at the…

I use my ipython terminal daily. It’s my go to way of running python most of the time. After you use it for a little bit you will probably want to setup a bit of your own configuration.

install ipython

Activate your virtual environment of choice and pip install it. Any time you are running your project in a virtual environment, you will need to install ipython inside it to access those packages from ipython.

pip install ipython

You are using a virtual environment right? Virtual environments like venv or conda can save you a ton of pain down the road.

profile_default

When…