百科.dev
全部条目在线工具AI 编程趋势榜开源项目技术资讯提交条目
登录
< 返回工具列表
M

ml-logger

> 编程语言
开源

用于 ML 项目的日志记录器、服务器和可视化仪表板

200 stars0 点赞0 次浏览
访问官网GitHub

工具介绍

用于 ML 项目的日志记录器、服务器和可视化仪表板

ML-Logger, A Simple and Scalable Logging Utility With a Beautiful Visualization Dashboard That Is Super Fast

ML-Logger makes it easy to:

  • save data locally and remotely, as binary, in a transparent pickle file, with the same API and zero configuration.
  • write from 500+ worker containers to a single instrumentation server
  • visualize matplotlib.pyplot figures from a remote server locally with logger.savefig('my_figure.png')

And ml-logger does all of these with minimal configuration — you can use the same logging code both locally and remotely with no code change.

ML-logger is highly performant -- the remote writes are asynchronous. For this reason it doesn't slow down your training even with 100+ metric keys.

Why did we built this, you might ask? Because we want to make it easy for people in ML to use the same logging code in all of they projects, so that it is easy to get started with someone else's baseline.

Getting Started!

To install ml_logger, do:

bash
pip install ml-logger

The landscape of python modules is a lot messier than that of javascript. The most up-to-date graphene requires the following versions:

bash
pip install sanic==20.9.0
pip install sanic-cors==0.10.0.post3
pip install sanic-graphql==1.1.0
yes | pip install graphene==2.1.3
yes | pip install graphql-core==2.1
yes | pip install graphql-relay==0.4.5
yes | pip install graphql-server-core==1.1.1

Add the following to your ~/.zshrc

bash
export ML_LOGGER_ROOT=
export ML_LOGGER_USER= # make sure it is the same on the cluster.

Now you can rock!

python
from ml_logger import logger

print(logger)
# ~> logging data to /tmp/ml-logger-debug

Log key/value pairs, and metrics:

python
for i in range(1):
    logger.log(metrics={'some_val/smooth': 10, 'status': f"step ({i})"}, reward=20, timestep=i)
    ### flush the data, otherwise the value would be overwritten with new values in the next iteration.
    logger.flush()
# outputs ~>
# ╒════════════════════╤════════════════════════════╕
# │       reward       │             20             │
# ├────────────────────┼────────────────────────────┤
...

Simple Logging Example

In your project files, do:

…

And it should print out

…

To View the Results

Note, the jwt based access control has not been implemented yet. So you should leave the token field empty.

Logging to a Server

Skip this if you just want to log locally. When training in parallel, you want to kickstart an logging server (Instrument Server). To do so, run:

bash
python -m ml_logger.server

Use ssh tunnel if you are running on a managed cluster (with SLURM for instance). Important: to set allow remote logging, you need to pass in --host=0.0.0.0 so that the server accepts non-localhost connections.

bash
python -m ml_logger.server --host=0.0.0.0

Asynchronously log the summary of LOTs of training metrics

A common scenario is you only want to upload averaged statistics of your metrics. A pattern that @jachiam uses is the following: store_metrics(), peak_stored_metrics(), and log_metrics_summary()

…

Table of Contents

  • logging matplotlib.pyplot figures on an headless server
  • [documentation under construction]

How to Develop

First clone repo, install dev dependencies, and install the module under evaluation mode.

bash
git clone https://github.com/episodeyang/ml_logger.git
cd ml_logger && cd ml_logger && pip install -r requirements-dev.txt
pip install -e .

Testing local-mode (without a server)

You should be inside ml_logger/ml_logger folder

bash
pwd # ~> ml_logger/ml_logger
make test

Testing with a server (You need to do both for an PR)

To test with a live server, first run (in a separate console)

python -m ml_logger.server --data-dir /tmp/ml-logger-debug

or do:

bash
make start-test-server

Then run this test script with the option:

bash
python -m pytest tests --capture=no --data-dir http://0.0.0.0:8081

or do

bash
make test-with-server

Your PR should have both of these two tests working. ToDo: add CI to this repo.

To Publish

You need twine, rst-lint etc, which are included in the requirements-dev.txt file.


Logging Matplotlib pyplots

Configuring The Experiment Folder

python
from ml_logger import logger, ML_Logger
from datetime import datetime

now = datetime.now()
logger.configure("/tmp/ml-logger-demo", "deep_Q_learning", f"{now:%Y%m%d-%H%M%S}")

This is a singleton pattern similar to matplotlib.pyplot. However, you could also use the logger constructor

python
logger = ML_Logger(root_dir="/tmp/ml-logger-demo", prefix=f"deep_Q_learning/{now:%Y%m%d-%H%M%S}")

Logging Text, and Metrics

python
logger.log({"some_var/smooth": 10}, some=Color(0.85, 'yellow', percent), step=3)

colored output: (where the values are yellow)

log
╒════════════════════╤════════════════════╕
│  some var/smooth   │         10         │
├────────────────────┼────────────────────┤
│        some        │       85.0%        │
╘════════════════════╧════════════════════╛

Logging Matplotlib Figures

We have optimized ML-Logger, so it supports any format that pyplot supports. To save a figure locally or remotely,

python
import matplotlib.pyplot as plt
import numpy as np

xs = np.linspace(-5, 5)

plt.plot(xs, np.cos(xs), label='Cosine Func')
logger.savefig('cosine_function.pdf')

Logging Videos

It is especially hard to visualize RL training sessions on a remote computer. With ML-Logger this is easy, and super fast. We optimized the serialization and transport process, so that a large stack of video tensor gets first compressed by ffmepg before getting sent over the wire.

The compression rate (and speed boost) can be 2000:1.

python
import numpy as np

def im(x, y):
    canvas = np.zeros((200, 200))
    for i in range(200):
        for j in range(200):
            if x - 5 )` is an http end point, 
the logger will instantiate a fast, future based logging client that launches 
http requests in a separate thread. We optimized the client so that it won't 
slow down your training code.

API wise, ML-logger makes it easy for you to log textual printouts, simple 
scalars, numpy tensors, image tensors, and `pyplot` figures. Because you might
also want to read data from the instrumentation server, we also made it possible to 
load numpy, pickle, text and binary files remotely.

In the future, we will start building an integrated dashboard with fast search, 
live figure update and markdown-based reporting/dashboarding to go with ml-logger.

Now give this a try, and profit!

Issues· 13 开放

查看全部 Issues在 GitHub 打开

暂无开放 Issues,或尚未同步最近议题。

> 标签

Python

暂无评论,来聊聊你的看法吧

> 工具信息

发布日期2026年8月1日
最后更新2026年9月18日
分类编程语言
定价开源

> 相关工具

T
TypeScript
JavaScript 的超集,为前端与全栈提供静态类型
P
Python
通用编程语言,广泛用于 Web、数据与 AI
G
Go
Google 推出的简洁高效系统语言