forked from arechen/EapilSkillMarket
Sync third-party and MCP marketplace plugins
Constraint: Public skills are published only by explicit administrator action unless they are tracked third-party market sources. Confidence: high Scope-risk: narrow Directive: Keep private/internal skills out of the public marketplace and preserve normal incremental market Git history. Tested: Marketplace validation passed.
This commit is contained in:
1 parent
cbde777e05
commit
9e74428671
15 files changed
+571
-42
No files matched your search
@@ -3,5 +3,5 @@
|
||||
"name": "playwright浏览器自动化操作",
|
||||
"version": "20260605",
|
||||
"keySource": "none",
|
||||
"syncedAt": "2026-10-10T16:02:37Z"
|
||||
"syncedAt": "2026-10-11T16:02:41Z"
|
||||
}
|
||||
@@ -2,8 +2,8 @@
|
||||
"sourceId": "next-skills",
|
||||
"repo": "https://github.com/vercel/next.js.git",
|
||||
"ref": "canary",
|
||||
"commit": "d8bfc3c994670c367e6e31853b78f42e8b7fba85",
|
||||
"commit": "d8c1c1b0002db8ccb63b58095f338e8bf423fa99",
|
||||
"adapter": "skill-collection",
|
||||
"sourcePath": "skills",
|
||||
"syncedAt": "2026-10-10T16:00:00Z"
|
||||
"syncedAt": "2026-10-11T16:00:00Z"
|
||||
}
|
||||
@@ -2,8 +2,8 @@
|
||||
"sourceId": "ppt-master",
|
||||
"repo": "https://github.com/hugohe3/ppt-master.git",
|
||||
"ref": "main",
|
||||
"commit": "b4efe3ddf237a97a42880164759162f8e2412f2f",
|
||||
"commit": "ca04346de145ed6462ddf34682848e71df5e20a2",
|
||||
"adapter": "claude-skill",
|
||||
"sourcePath": "skills/ppt-master",
|
||||
"syncedAt": "2026-10-08T15:59:59Z"
|
||||
"syncedAt": "2026-10-11T16:00:00Z"
|
||||
}
|
||||
@@ -24,6 +24,14 @@ unified `source_to_md.py` dispatcher writes a missing profile before success.
|
||||
| Image manifest | `<stem>_files/image_manifest.json` when image metadata is available |
|
||||
| Conversion profile | `<stem>.conversion_profile.json` beside the Markdown output when written |
|
||||
|
||||
For multi-file or directory inputs, the unified entry and direct PDF, document,
|
||||
Excel, and PowerPoint backends allocate distinct Markdown paths within the run.
|
||||
Without `-o`, each output stays in its source's directory; with `-o`, outputs
|
||||
share that directory. Collisions with inputs or other planned outputs gain
|
||||
`_2`, `_3`, etc., reported as `[INFO] Renamed output ...` on stderr. Assets and
|
||||
profiles follow the chosen Markdown stem. Separate runs retain the existing
|
||||
overwrite behavior for unrelated outputs.
|
||||
|
||||
When present, the conversion profile is metadata only: converter, source path,
|
||||
Markdown structure counts, asset directory, image manifest path, and image
|
||||
count. Downstream PPT workflows still use Markdown and the image manifest as
|
||||
@@ -76,8 +84,13 @@ Useful options:
|
||||
refuses an existing file except its own Markdown/text passthrough.
|
||||
|
||||
For multi-source project intake, use `project_manager.py import-sources` with
|
||||
all source paths / URLs. For local files, the default is to keep generated
|
||||
Markdown and profile outputs beside the original source.
|
||||
all source paths / URLs. Same-stem convertible sources receive distinct
|
||||
archive names such as `budget_2.xlsm`, so their Markdown, profiles, assets, and
|
||||
PPTX intake artifacts stay separate. Pre-existing Markdown and explicitly
|
||||
supplied same-stem Markdown still suppress automatic conversion; PPTX intake
|
||||
remains separate for each archived deck. See
|
||||
[`project.md`](project.md). For standalone local conversion, the default is to
|
||||
keep generated Markdown and profiles beside the original source.
|
||||
`source_to_md.py` and the backend converters support single files, explicit
|
||||
multi-file inputs, and non-recursive directory inputs.
|
||||
|
||||
@@ -124,7 +137,8 @@ Native path (no external binary required):
|
||||
- `.docx` — via `mammoth`; text-only tables (with footnotes) and chart data become pipe Markdown, and OMML / Office Math equations (Word-native or MathType "Convert to Office Math") are rewritten to inline LaTeX. Classic MathType OLE objects carry no OMML and are kept only as their preview image.
|
||||
- `.html` / `.htm` — via `markdownify` + `beautifulsoup4`
|
||||
- `.epub` — via `ebooklib` + `markdownify`
|
||||
- `.ipynb` — via `nbconvert`
|
||||
- `.ipynb` — via `nbconvert`; code-cell output images and Markdown attachments
|
||||
use `<output-stem>_files/` with relative Markdown references
|
||||
|
||||
Pandoc fallback (only if you need these):
|
||||
- `.doc`, `.odt`, `.rtf`, `.tex`/`.latex`, `.rst`, `.org`, `.typ`
|
||||
@@ -155,9 +169,10 @@ pip install mammoth markdownify ebooklib nbconvert beautifulsoup4
|
||||
# Windows: https://pandoc.org/installing.html
|
||||
```
|
||||
|
||||
All paths produce `<input>.md`. Extracted assets use a sibling `<input>_files/`
|
||||
By default, each input produces `<input-stem>.md`, subject to batch collision
|
||||
renaming. Extracted assets use a sibling `<output-stem>_files/`
|
||||
directory with relative references. Without assets, that directory need not remain.
|
||||
The direct backend then attempts `<input>.conversion_profile.json` under the
|
||||
The direct backend then attempts `<output-stem>.conversion_profile.json` under the
|
||||
shared best-effort sidecar contract.
|
||||
|
||||
## `source_to_md/excel_to_md.py`
|
||||
@@ -250,7 +265,7 @@ Outputs (per source deck, prefixed by file stem):
|
||||
- `<stem>.slide_library.json` — text slots, geometry, native tables, native chart display caches, and SmartArt nodes/connections
|
||||
- `source_profile.json` — the single multi-deck index: a compact Strategist-facing digest per deck (over identity, tables, charts, SmartArt, and page types) under `decks[]`, with prefixed artifact pointers
|
||||
|
||||
`project_manager.py import-sources` runs this automatically for PPTX/PPTM/PPSX/PPSM/POTX/POTM inputs and stores the bundle directly under `analysis/`. Multi-deck per project: importing several PPTX files gives each its own `<stem>.*` artifacts and a `decks[]` entry in the shared `source_profile.json` index (re-importing the same stem replaces its entry). The beautify profile stays single-deck and reads one chosen deck's `<stem>.*` artifacts.
|
||||
`project_manager.py import-sources` runs this automatically for PPTX/PPTM/PPSX/PPSM/POTX/POTM inputs and stores the bundle directly under `analysis/`. Multi-deck per project: each deck uses its archived stem, including any collision suffix, for `<stem>.*` artifacts and its `decks[]` entry in the shared `source_profile.json` index. Re-running intake for an existing archived stem replaces its entry. The beautify profile stays single-deck and reads one chosen deck's `<stem>.*` artifacts.
|
||||
|
||||
Usage boundary:
|
||||
- Standard generation uses these fields as facts and recommendation candidates; it does not inherit source slide coordinates or page order by default.
|
||||
|
||||
@@ -58,6 +58,11 @@ Notes:
|
||||
pass the source file/directory once when generated Markdown lives beside the
|
||||
original source. If Step 1 used `-o` to write Markdown elsewhere, pass both
|
||||
the original source path/directory and the Markdown output path/directory.
|
||||
- Distinct same-stem convertible sources in one `import-sources` call
|
||||
receive archive suffixes (`budget.xlsm` → `budget_2.xlsm`). Markdown, conversion
|
||||
profiles, `_files/` assets, and PPTX analysis follow the archived stem.
|
||||
Pre-existing Markdown and explicitly supplied same-stem Markdown still
|
||||
suppress automatic conversion; PPTX intake still keeps each deck separate.
|
||||
- A projects-local supplied source directory left strictly empty after import
|
||||
(or empty from the start) is removed; every directory outside `projects/`
|
||||
remains untouched. `--copy` never removes directories.
|
||||
@@ -101,8 +106,9 @@ Notes:
|
||||
- PPTX-family inputs are enriched automatically under `analysis/` with
|
||||
per-deck `<stem>.identity.json` / `<stem>.slide_library.json` plus the shared
|
||||
multi-deck index `source_profile.json` (`decks[]`).
|
||||
Multi-deck per project: several PPTX imports each get their own `<stem>.*`
|
||||
artifacts and a `decks[]` entry; re-importing the same stem replaces its entry.
|
||||
Multi-deck per project: `<stem>` is the archived stem, including any collision
|
||||
suffix. Each deck gets its own artifacts and `decks[]` entry; re-running
|
||||
intake for an existing archived stem replaces its entry.
|
||||
|
||||
### On-demand page execution view
|
||||
|
||||
|
||||
+43
-9
@@ -820,6 +820,8 @@ class ProjectManager:
|
||||
raise ValueError("At least one source path or URL is required")
|
||||
|
||||
sources_dir = self._source_dir(project_dir)
|
||||
existing_markdown = {path.resolve() for path in sources_dir.glob("*.md")}
|
||||
reserved_markdown: set[Path] = set()
|
||||
summary: dict[str, list[str]] = {
|
||||
"archived": [],
|
||||
"url_records": [],
|
||||
@@ -998,12 +1000,48 @@ class ProjectManager:
|
||||
summary["notes"].append(note)
|
||||
continue
|
||||
|
||||
archive_target = sources_dir / source_path.name
|
||||
if source_path.resolve() != archive_target.resolve():
|
||||
archive_target = self._ensure_unique_path(archive_target)
|
||||
canonical_markdown_path = archive_target.with_suffix(".md")
|
||||
has_explicit_markdown = archive_target.stem in explicit_markdown_stems
|
||||
if suffix in PDF_SUFFIXES | PRESENTATION_SUFFIXES | EXCEL_SUFFIXES | DOC_SUFFIXES:
|
||||
markdown_target = canonical_markdown_path
|
||||
keep_markdown = (
|
||||
has_explicit_markdown or markdown_target.resolve() in existing_markdown
|
||||
)
|
||||
if not keep_markdown or markdown_target.resolve() in reserved_markdown:
|
||||
# Reserve the source stem before conversion, including failed
|
||||
# conversions that may already have written intake or assets.
|
||||
stem = archive_target.stem
|
||||
counter = 2
|
||||
while (
|
||||
markdown_target.resolve() in reserved_markdown
|
||||
or markdown_target.exists()
|
||||
or archive_target.stem in explicit_markdown_stems
|
||||
or (archive_target.exists() and archive_target.resolve() != source_path.resolve())
|
||||
or markdown_target.with_suffix(".conversion_profile.json").exists()
|
||||
or (sources_dir / f"{archive_target.stem}_files").exists()
|
||||
or (project_dir / "analysis" / f"{archive_target.stem}.identity.json").exists()
|
||||
or (project_dir / "analysis" / f"{archive_target.stem}.slide_library.json").exists()
|
||||
):
|
||||
archive_target = archive_target.with_name(f"{stem}_{counter}{source_path.suffix}")
|
||||
markdown_target = archive_target.with_suffix(".md")
|
||||
counter += 1
|
||||
# PPTX intake still writes analysis when Markdown is reused.
|
||||
reserved_markdown.add(markdown_target.resolve())
|
||||
if not keep_markdown:
|
||||
canonical_markdown_path = markdown_target
|
||||
archived_path = self._copy_or_move_file(
|
||||
source_path,
|
||||
sources_dir / source_path.name,
|
||||
archive_target,
|
||||
move=effective_move,
|
||||
)
|
||||
summary["archived"].append(str(archived_path))
|
||||
if archived_path.name != source_path.name:
|
||||
summary["notes"].append(
|
||||
f"{item}: archived as {archived_path.name} to avoid a filename or source-stem collision"
|
||||
)
|
||||
|
||||
if suffix in BITMAP_IMAGE_SUFFIXES:
|
||||
images_dir = project_dir / "images"
|
||||
@@ -1017,8 +1055,7 @@ class ProjectManager:
|
||||
"to avoid a filename collision"
|
||||
)
|
||||
elif suffix in PDF_SUFFIXES:
|
||||
canonical_markdown_path = sources_dir / f"{archived_path.stem}.md"
|
||||
if archived_path.stem in explicit_markdown_stems:
|
||||
if has_explicit_markdown:
|
||||
summary["notes"].append(
|
||||
f"{item}: skipped PDF auto-conversion because a same-stem Markdown source was provided"
|
||||
)
|
||||
@@ -1040,7 +1077,6 @@ class ProjectManager:
|
||||
except Exception as exc: # pragma: no cover - summary path
|
||||
summary["skipped"].append(f"{item}: PDF conversion failed ({exc})")
|
||||
elif suffix in PRESENTATION_SUFFIXES:
|
||||
canonical_markdown_path = sources_dir / f"{archived_path.stem}.md"
|
||||
try:
|
||||
intake_dir = self._import_pptx_intake(archived_path, project_dir)
|
||||
intake_str = str(intake_dir)
|
||||
@@ -1048,7 +1084,7 @@ class ProjectManager:
|
||||
summary["analysis"].append(intake_str)
|
||||
except Exception as exc: # pragma: no cover - summary path
|
||||
summary["notes"].append(f"{item}: PPTX intake analysis failed ({exc})")
|
||||
if archived_path.stem in explicit_markdown_stems:
|
||||
if has_explicit_markdown:
|
||||
summary["notes"].append(
|
||||
f"{item}: skipped presentation auto-conversion because a same-stem Markdown source was provided"
|
||||
)
|
||||
@@ -1070,8 +1106,7 @@ class ProjectManager:
|
||||
except Exception as exc: # pragma: no cover - summary path
|
||||
summary["skipped"].append(f"{item}: presentation conversion failed ({exc})")
|
||||
elif suffix in EXCEL_SUFFIXES:
|
||||
canonical_markdown_path = sources_dir / f"{archived_path.stem}.md"
|
||||
if archived_path.stem in explicit_markdown_stems:
|
||||
if has_explicit_markdown:
|
||||
summary["notes"].append(
|
||||
f"{item}: skipped Excel auto-conversion because a same-stem Markdown source was provided"
|
||||
)
|
||||
@@ -1102,8 +1137,7 @@ class ProjectManager:
|
||||
f"{item}: archived as a plain-text table source; no Markdown conversion needed"
|
||||
)
|
||||
elif suffix in DOC_SUFFIXES:
|
||||
canonical_markdown_path = sources_dir / f"{archived_path.stem}.md"
|
||||
if archived_path.stem in explicit_markdown_stems:
|
||||
if has_explicit_markdown:
|
||||
summary["notes"].append(
|
||||
f"{item}: skipped document auto-conversion because a same-stem Markdown source was provided"
|
||||
)
|
||||
|
||||
@@ -84,8 +84,16 @@ def _output_for(
|
||||
batch_mode: bool,
|
||||
used_outputs: set[Path],
|
||||
) -> Path:
|
||||
if output_arg and batch_mode:
|
||||
return unique_output_path(Path(output_arg), source.stem, used_outputs)
|
||||
if batch_mode:
|
||||
output_dir = Path(output_arg) if output_arg else source.parent
|
||||
default_output = output_dir / f"{source.stem}.md"
|
||||
output = unique_output_path(output_dir, source.stem, used_outputs)
|
||||
if output != default_output:
|
||||
_print_status(
|
||||
f"[INFO] Renamed output for {source}: {default_output} -> {output} "
|
||||
"(input/output collision)"
|
||||
)
|
||||
return output
|
||||
if output_arg:
|
||||
return Path(output_arg)
|
||||
return source.with_suffix(".md")
|
||||
@@ -115,7 +123,7 @@ def run_path_batch(
|
||||
success_count = 0
|
||||
failed: list[str] = []
|
||||
skipped: list[str] = list(expansion_errors)
|
||||
used_outputs: set[Path] = set()
|
||||
used_outputs = {source.resolve() for source in sources}
|
||||
|
||||
for source in sources:
|
||||
output = _output_for(source, output_arg, batch_mode, used_outputs)
|
||||
|
||||
+15
-5
@@ -1498,18 +1498,23 @@ def _convert_ipynb(input_file: Path, out_file: Path) -> str:
|
||||
print("[ERROR] nbconvert not installed. Run: pip install nbconvert")
|
||||
return ""
|
||||
|
||||
# Pre-process cell-level markdown attachments: nbconvert leaves
|
||||
# `attachment:<name>` references intact but doesn't write the files.
|
||||
# Extract them into our outputs dict so FilesWriter picks them up.
|
||||
# Extract attachments once with notebook-wide names; nbconvert's own
|
||||
# attachment extractor can reuse names from different Markdown cells.
|
||||
nb = nbformat.read(str(input_file), as_version=4)
|
||||
extra_outputs: dict[str, bytes] = {}
|
||||
rel_media_dir = f"{out_file.stem}_files"
|
||||
output_stems = {
|
||||
Path(output.metadata["filename"]).stem.casefold()
|
||||
for cell in nb.cells
|
||||
for output in cell.get("outputs", [])
|
||||
if output.get("metadata", {}).get("filename")
|
||||
}
|
||||
|
||||
attach_counter = 0
|
||||
for cell in nb.cells:
|
||||
if cell.cell_type != "markdown":
|
||||
continue
|
||||
attachments = getattr(cell, "attachments", None) or {}
|
||||
attachments = cell.pop("attachments", None) or {}
|
||||
if not attachments:
|
||||
continue
|
||||
for att_name, mime_data in attachments.items():
|
||||
@@ -1519,6 +1524,9 @@ def _convert_ipynb(input_file: Path, out_file: Path) -> str:
|
||||
if ext == ".jpe":
|
||||
ext = ".jpg"
|
||||
filename = f"attachment_{attach_counter:03d}{ext}"
|
||||
while Path(filename).stem.casefold() in output_stems:
|
||||
attach_counter += 1
|
||||
filename = f"attachment_{attach_counter:03d}{ext}"
|
||||
out_path = f"{rel_media_dir}/{filename}"
|
||||
try:
|
||||
extra_outputs[out_path] = base64.b64decode(b64)
|
||||
@@ -1530,7 +1538,9 @@ def _convert_ipynb(input_file: Path, out_file: Path) -> str:
|
||||
cell.source = src
|
||||
|
||||
exporter = MarkdownExporter()
|
||||
body, resources = exporter.from_notebook_node(nb)
|
||||
body, resources = exporter.from_notebook_node(
|
||||
nb, resources={"output_files_dir": rel_media_dir},
|
||||
)
|
||||
|
||||
# Merge attachment outputs with whatever nbconvert collected
|
||||
resources.setdefault("outputs", {}).update(extra_outputs)
|
||||
|
||||
+221
@@ -13,6 +13,7 @@ Dependencies:
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import base64
|
||||
import io
|
||||
import json
|
||||
import socket
|
||||
@@ -412,6 +413,226 @@ class ArchivedProjectImportTests(unittest.TestCase):
|
||||
)
|
||||
|
||||
|
||||
class SameStemProjectImportTests(unittest.TestCase):
|
||||
def setUp(self) -> None:
|
||||
self.temporary = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(self.temporary.cleanup)
|
||||
self.root = Path(self.temporary.name)
|
||||
self.manager = project_cli.ProjectManager(base_dir=self.root / "projects")
|
||||
|
||||
def _project(self, name="target") -> Path:
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
return Path(self.manager.init_project(name, quick_generate=True))
|
||||
|
||||
def _workbooks(self, root: Path) -> list[Path]:
|
||||
from openpyxl import Workbook
|
||||
|
||||
root.mkdir(parents=True)
|
||||
sources = []
|
||||
for suffix, total in ((".xlsx", 20000), (".xlsm", 22500)):
|
||||
workbook = Workbook()
|
||||
workbook.active.append(["Category", "Amount"])
|
||||
workbook.active.append(["Total", total])
|
||||
source = root / f"budget{suffix}"
|
||||
workbook.save(source)
|
||||
workbook.close()
|
||||
sources.append(source)
|
||||
return sources
|
||||
|
||||
def _import(self, project: Path, sources: list[Path]) -> dict:
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
return self.manager.import_sources(str(project), list(map(str, sources)), copy=True)
|
||||
|
||||
def _assert_conversions(self, project: Path, summary: dict, labels: list[str]) -> None:
|
||||
self.assertEqual(summary["skipped"], [])
|
||||
self.assertEqual(len(summary["markdown"]), len(labels))
|
||||
self.assertEqual(len(set(summary["markdown"])), len(labels))
|
||||
for raw_path, markdown_path, label in zip(summary["archived"], summary["markdown"], labels):
|
||||
source = Path(raw_path)
|
||||
markdown = Path(markdown_path)
|
||||
self.assertEqual(source.stem, markdown.stem)
|
||||
content = markdown.read_text(encoding="utf-8").replace(r"\-", "-")
|
||||
self.assertIn(label, content)
|
||||
for other_label in labels:
|
||||
if other_label != label:
|
||||
self.assertNotIn(other_label, content)
|
||||
profile = json.loads(profile_path_for(markdown).read_text(encoding="utf-8"))
|
||||
self.assertEqual(profile["source"]["name"], source.name)
|
||||
self.assertEqual(profile["outputs"]["markdown"], markdown.name)
|
||||
|
||||
def test_same_stem_workbooks_each_convert_with_matching_archived_names(self) -> None:
|
||||
for reverse in (False, True):
|
||||
with self.subTest(reverse=reverse):
|
||||
sources = self._workbooks(self.root / f"inputs_{reverse}")
|
||||
labels = ["20000", "22500"]
|
||||
if reverse:
|
||||
sources.reverse()
|
||||
labels.reverse()
|
||||
originals = {source: source.read_bytes() for source in sources}
|
||||
project = self._project(f"target_{reverse}")
|
||||
summary = self._import(project, sources)
|
||||
self._assert_conversions(project, summary, labels)
|
||||
self.assertEqual([Path(path).name for path in summary["markdown"]],
|
||||
["budget.md", "budget_2.md"])
|
||||
for source, archived in zip(sources, summary["archived"]):
|
||||
self.assertEqual(Path(archived).read_bytes(), originals[source])
|
||||
self.assertEqual(source.read_bytes(), originals[source])
|
||||
|
||||
def test_document_and_presentation_assets_and_intake_keep_each_source(self) -> None:
|
||||
from docx import Document
|
||||
from PIL import Image
|
||||
from pptx import Presentation
|
||||
from pptx.util import Inches
|
||||
|
||||
for kind in ("document", "presentation"):
|
||||
with self.subTest(kind=kind):
|
||||
inputs = self.root / kind
|
||||
inputs.mkdir()
|
||||
sources = []
|
||||
expected_images = []
|
||||
labels = ["FIRST-SOURCE-20000", "SECOND-SOURCE-22500"]
|
||||
for index, color in enumerate(("red", "blue")):
|
||||
buffer = io.BytesIO()
|
||||
Image.new("RGB", (16, 16), color).save(buffer, format="PNG")
|
||||
raw = buffer.getvalue()
|
||||
expected_images.append(raw)
|
||||
if kind == "document":
|
||||
source = inputs / ("report.docx" if index == 0 else "report.html")
|
||||
if index == 0:
|
||||
document = Document()
|
||||
document.add_paragraph(labels[index])
|
||||
document.add_picture(io.BytesIO(raw))
|
||||
document.save(source)
|
||||
else:
|
||||
encoded = base64.b64encode(raw).decode("ascii")
|
||||
source.write_text(
|
||||
f'<p>{labels[index]}</p><img src="data:image/png;base64,{encoded}">',
|
||||
encoding="utf-8",
|
||||
)
|
||||
else:
|
||||
source = inputs / ("deck.pptx" if index == 0 else "deck.potx")
|
||||
presentation = Presentation()
|
||||
slide = presentation.slides.add_slide(presentation.slide_layouts[6])
|
||||
slide.shapes.add_textbox(Inches(1), Inches(1), Inches(5), Inches(1)).text = labels[index]
|
||||
slide.shapes.add_picture(io.BytesIO(raw), Inches(1), Inches(2))
|
||||
presentation.save(source)
|
||||
sources.append(source)
|
||||
project = self._project(kind)
|
||||
summary = self._import(project, sources)
|
||||
self._assert_conversions(project, summary, labels)
|
||||
stems = {Path(path).stem for path in summary["archived"]}
|
||||
manifest = json.loads((project / "images/image_manifest.json").read_text(encoding="utf-8"))
|
||||
self.assertEqual({row["source_namespace"] for row in manifest}, stems)
|
||||
self.assertEqual({(project / "images" / row["filename"]).read_bytes() for row in manifest},
|
||||
set(expected_images))
|
||||
for source, raw in zip(summary["archived"], expected_images):
|
||||
assets = project / "sources" / f"{Path(source).stem}_files"
|
||||
self.assertEqual(len(list(assets.glob("*.png"))), 1)
|
||||
self.assertEqual(next(assets.glob("*.png")).read_bytes(), raw)
|
||||
if kind == "presentation":
|
||||
index = json.loads((project / "analysis/source_profile.json").read_text(encoding="utf-8"))
|
||||
self.assertEqual(index["deck_count"], 2)
|
||||
self.assertEqual({deck["stem"] for deck in index["decks"]}, stems)
|
||||
self.assertEqual({deck["source_pptx"] for deck in index["decks"]}, set(summary["archived"]))
|
||||
for stem, label in zip(("deck", "deck_2"), labels):
|
||||
self.assertTrue((project / "analysis" / f"{stem}.identity.json").is_file())
|
||||
library = project / "analysis" / f"{stem}.slide_library.json"
|
||||
self.assertIn(label, library.read_text(encoding="utf-8"))
|
||||
|
||||
def test_reused_markdown_does_not_merge_same_stem_presentation_intake(self) -> None:
|
||||
from pptx import Presentation
|
||||
from pptx.util import Inches
|
||||
|
||||
for explicit in (False, True):
|
||||
with self.subTest(explicit=explicit):
|
||||
inputs = self.root / f"companion_{explicit}"
|
||||
inputs.mkdir()
|
||||
sources = []
|
||||
labels = ["FIRST-SOURCE-20000", "SECOND-SOURCE-22500"]
|
||||
for suffix, label in zip((".pptx", ".potx"), labels):
|
||||
presentation = Presentation()
|
||||
slide = presentation.slides.add_slide(presentation.slide_layouts[6])
|
||||
slide.shapes.add_textbox(Inches(1), Inches(1), Inches(5), Inches(1)).text = label
|
||||
source = inputs / f"deck{suffix}"
|
||||
presentation.save(source)
|
||||
sources.append(source)
|
||||
project = self._project(f"companion_{explicit}")
|
||||
(project / "sources").mkdir()
|
||||
markdown = (inputs if explicit else project / "sources") / "deck.md"
|
||||
markdown.write_text("# Authoritative Markdown\n", encoding="utf-8")
|
||||
summary = self._import(project, [*sources, markdown] if explicit else sources)
|
||||
self.assertEqual(summary["skipped"], [])
|
||||
self.assertEqual([Path(path).name for path in summary["archived"][:2]],
|
||||
["deck.pptx", "deck_2.potx"])
|
||||
self.assertEqual((project / "sources/deck.md").read_text(encoding="utf-8"),
|
||||
"# Authoritative Markdown\n")
|
||||
self.assertEqual(list((project / "sources").glob("*.conversion_profile.json")), [])
|
||||
self.assertEqual(sum("skipped presentation auto-conversion" in note
|
||||
for note in summary["notes"]), 2)
|
||||
index = json.loads((project / "analysis/source_profile.json").read_text(encoding="utf-8"))
|
||||
self.assertEqual(index["deck_count"], 2)
|
||||
self.assertEqual({deck["stem"] for deck in index["decks"]}, {"deck", "deck_2"})
|
||||
self.assertEqual({deck["source_pptx"] for deck in index["decks"]},
|
||||
set(summary["archived"][:2]))
|
||||
for stem, label, other_label in zip(("deck", "deck_2"), labels, reversed(labels)):
|
||||
self.assertTrue((project / "analysis" / f"{stem}.identity.json").is_file())
|
||||
library = (project / "analysis" / f"{stem}.slide_library.json").read_text(encoding="utf-8")
|
||||
self.assertIn(label, library)
|
||||
self.assertNotIn(other_label, library)
|
||||
|
||||
def test_explicit_markdown_still_suppresses_both_conversions(self) -> None:
|
||||
for markdown_first in (False, True):
|
||||
with self.subTest(markdown_first=markdown_first):
|
||||
sources = self._workbooks(self.root / f"explicit_{markdown_first}")
|
||||
markdown = sources[0].with_suffix(".md")
|
||||
markdown.write_text("# Authoritative Markdown\n", encoding="utf-8")
|
||||
inputs = [markdown, *sources] if markdown_first else [*sources, markdown]
|
||||
project = self._project(f"explicit_{markdown_first}")
|
||||
summary = self._import(project, inputs)
|
||||
self.assertEqual(summary["skipped"], [])
|
||||
self.assertEqual((project / "sources/budget.md").read_text(encoding="utf-8"),
|
||||
"# Authoritative Markdown\n")
|
||||
self.assertEqual(list((project / "sources").glob("*.conversion_profile.json")), [])
|
||||
self.assertEqual(sum("same-stem Markdown source was provided" in note
|
||||
for note in summary["notes"]), 2)
|
||||
|
||||
def test_markdown_existing_before_import_is_reused(self) -> None:
|
||||
sources = self._workbooks(self.root / "existing")
|
||||
project = self._project()
|
||||
(project / "sources").mkdir()
|
||||
markdown = project / "sources/budget.md"
|
||||
markdown.write_text("# Previous conversion\n", encoding="utf-8")
|
||||
summary = self._import(project, sources)
|
||||
self.assertEqual(summary["markdown"], [str(markdown), str(markdown)])
|
||||
self.assertEqual(markdown.read_text(encoding="utf-8"), "# Previous conversion\n")
|
||||
self.assertEqual(sum("budget.md already exists" in note for note in summary["notes"]), 2)
|
||||
self.assertEqual(list((project / "sources").glob("*.conversion_profile.json")), [])
|
||||
|
||||
def test_generated_suffix_avoids_existing_markdown_and_source_names(self) -> None:
|
||||
sources = self._workbooks(self.root / "suffixes")
|
||||
project = self._project()
|
||||
(project / "sources").mkdir()
|
||||
existing = project / "sources/budget_2.md"
|
||||
existing.write_text("# Unrelated existing source\n", encoding="utf-8")
|
||||
occupied = project / "sources/budget_3.xlsm"
|
||||
occupied.write_bytes(b"archived original")
|
||||
summary = self._import(project, sources)
|
||||
self._assert_conversions(project, summary, ["20000", "22500"])
|
||||
self.assertEqual([Path(path).name for path in summary["markdown"]],
|
||||
["budget.md", "budget_4.md"])
|
||||
self.assertEqual(existing.read_text(encoding="utf-8"), "# Unrelated existing source\n")
|
||||
self.assertEqual(occupied.read_bytes(), b"archived original")
|
||||
|
||||
def test_failed_conversion_still_reserves_its_source_stem(self) -> None:
|
||||
sources = self._workbooks(self.root / "failure")
|
||||
sources[0].write_bytes(b"invalid workbook")
|
||||
project = self._project()
|
||||
summary = self._import(project, sources)
|
||||
self.assertEqual(len(summary["skipped"]), 1)
|
||||
self.assertEqual([Path(path).name for path in summary["markdown"]], ["budget_2.md"])
|
||||
self.assertIn("22500", Path(summary["markdown"][0]).read_text(encoding="utf-8"))
|
||||
|
||||
|
||||
class _CurlHeaders(dict):
|
||||
"""Represent repeated curl headers without importing optional curl_cffi."""
|
||||
|
||||
|
||||
+67
@@ -271,6 +271,73 @@ class SourceCollisionTests(unittest.TestCase):
|
||||
for line in result.stdout.splitlines() if line.startswith("{")
|
||||
]
|
||||
|
||||
def test_direct_pdf_batch_preserves_inputs_and_reports_renames(self) -> None:
|
||||
try:
|
||||
import fitz
|
||||
except ImportError:
|
||||
self.skipTest("PyMuPDF unavailable")
|
||||
for reverse in (False, True):
|
||||
for separate_output in (False, True):
|
||||
with self.subTest(reverse=reverse, output=separate_output):
|
||||
root = self.root / f"pdf_{reverse}_{separate_output}"
|
||||
root.mkdir()
|
||||
sources = [root / "same.pdf", root / "same.md", root / "same_2.md"]
|
||||
originals = {}
|
||||
for index, source in enumerate(sources):
|
||||
with fitz.open() as document:
|
||||
page = document.new_page()
|
||||
page.insert_text((72, 150), f"PDF-SOURCE-{index}")
|
||||
originals[source] = document.tobytes()
|
||||
source.write_bytes(originals[source])
|
||||
if reverse:
|
||||
sources.reverse()
|
||||
arguments = [str(path) for path in sources]
|
||||
if separate_output:
|
||||
arguments += ["-o", str(root)]
|
||||
result = subprocess.run(
|
||||
[sys.executable, str(WEB_BACKEND_DIR / "pdf_to_md.py"),
|
||||
*arguments, "--images", "none"],
|
||||
capture_output=True, text=True, encoding="utf-8", check=False,
|
||||
)
|
||||
self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
|
||||
for source, content in originals.items():
|
||||
self.assertEqual(source.read_bytes(), content)
|
||||
profiles = sorted(root.glob("*.conversion_profile.json"))
|
||||
self.assertEqual(len(profiles), 3)
|
||||
for path in profiles:
|
||||
profile = json.loads(path.read_text(encoding="utf-8"))
|
||||
source = root / profile["source"]["name"]
|
||||
output = root / profile["outputs"]["markdown"]
|
||||
self.assertNotIn(output, sources)
|
||||
self.assertIn(
|
||||
f"[INFO] Renamed output for {source}: {source.with_suffix('.md')} "
|
||||
f"-> {output} (input/output collision)", result.stderr,
|
||||
)
|
||||
|
||||
def test_direct_batch_keeps_outputs_in_each_source_directory(self) -> None:
|
||||
from openpyxl import Workbook
|
||||
|
||||
sources = []
|
||||
for name in ("first", "second"):
|
||||
directory = self.root / name
|
||||
directory.mkdir()
|
||||
source = directory / "same.xlsx"
|
||||
workbook = Workbook()
|
||||
workbook.active.append([name])
|
||||
workbook.save(source)
|
||||
workbook.close()
|
||||
sources.append(source)
|
||||
for _ in range(2):
|
||||
result = subprocess.run(
|
||||
[sys.executable, str(WEB_BACKEND_DIR / "excel_to_md.py"), *map(str, sources)],
|
||||
capture_output=True, text=True, encoding="utf-8", check=False,
|
||||
)
|
||||
self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
|
||||
self.assertNotIn("Renamed output", result.stderr)
|
||||
for source in sources:
|
||||
self.assertEqual(list(source.parent.glob("*.md")), [source.with_suffix(".md")])
|
||||
self.assertIn(source.parent.name, source.with_suffix(".md").read_text(encoding="utf-8"))
|
||||
|
||||
def test_same_stem_batch_preserves_markdown_in_both_orders(self) -> None:
|
||||
for reverse in (False, True):
|
||||
for output_directory in (False, True):
|
||||
|
||||
+125
@@ -5,8 +5,11 @@ Usage: python3 -m unittest tests.test_source_to_md_content_loss_docx
|
||||
Dependencies: python-docx, mammoth
|
||||
"""
|
||||
|
||||
import base64
|
||||
import io
|
||||
import json
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
@@ -26,6 +29,128 @@ sys.path.insert(0, str(SCRIPTS_DIR / "source_to_md"))
|
||||
import doc_to_md # noqa: E402
|
||||
|
||||
|
||||
class NotebookContentLossTests(unittest.TestCase):
|
||||
def test_attachments_do_not_overwrite_named_outputs_or_each_other(self) -> None:
|
||||
try:
|
||||
import nbconvert # noqa: F401
|
||||
import nbformat
|
||||
except ImportError:
|
||||
self.skipTest("optional nbconvert/nbformat unavailable")
|
||||
from PIL import Image
|
||||
|
||||
for filename in ("attachment_001", "attachment_001.png"):
|
||||
with self.subTest(filename=filename), tempfile.TemporaryDirectory() as tmp:
|
||||
root = Path(tmp)
|
||||
expected = []
|
||||
encoded = []
|
||||
for color in ("red", "blue", "green"):
|
||||
buffer = io.BytesIO()
|
||||
Image.new("RGB", (8, 8), color).save(buffer, format="PNG")
|
||||
expected.append(buffer.getvalue())
|
||||
encoded.append(base64.b64encode(buffer.getvalue()).decode("ascii"))
|
||||
notebook = nbformat.v4.new_notebook(cells=[
|
||||
nbformat.v4.new_code_cell("# Named output", outputs=[
|
||||
nbformat.v4.new_output(
|
||||
"display_data", data={"image/png": encoded[0]},
|
||||
metadata={"filename": filename},
|
||||
),
|
||||
]),
|
||||
*[
|
||||
nbformat.v4.new_markdown_cell(
|
||||
"",
|
||||
attachments={"picture.png": {"image/png": data}},
|
||||
)
|
||||
for data in encoded[1:]
|
||||
],
|
||||
])
|
||||
source = root / "book.ipynb"
|
||||
nbformat.write(notebook, source)
|
||||
with redirect_stdout(io.StringIO()):
|
||||
markdown = doc_to_md.convert_to_markdown(str(source))
|
||||
refs = re.findall(r"!\[[^\]]*\]\(([^)]+)\)", markdown)
|
||||
self.assertEqual(len(refs), 3)
|
||||
self.assertEqual(len(set(refs)), 3)
|
||||
for ref, raw in zip(refs, expected):
|
||||
self.assertEqual(Path(ref).parent, Path("book_files"))
|
||||
self.assertEqual((root / ref).read_bytes(), raw)
|
||||
manifest = json.loads((root / "book_files/image_manifest.json").read_text(encoding="utf-8"))
|
||||
self.assertEqual({row["filename"] for row in manifest}, {Path(ref).name for ref in refs})
|
||||
self.assertTrue(all(len(row["occurrences"]) == 1 for row in manifest))
|
||||
|
||||
def test_batch_images_and_attachments_remain_independent_after_import(self) -> None:
|
||||
try:
|
||||
import nbconvert # noqa: F401
|
||||
import nbformat
|
||||
except ImportError:
|
||||
self.skipTest("optional nbconvert/nbformat unavailable")
|
||||
from PIL import Image
|
||||
from project_management.cli import ProjectManager
|
||||
|
||||
for same_stem in (False, True):
|
||||
with self.subTest(same_stem=same_stem), tempfile.TemporaryDirectory() as tmp:
|
||||
root = Path(tmp)
|
||||
sources = []
|
||||
expected = []
|
||||
for index, color in enumerate(("red", "blue")):
|
||||
source_dir = root / str(index)
|
||||
source_dir.mkdir()
|
||||
source = source_dir / ("book.ipynb" if same_stem else f"book{index}.ipynb")
|
||||
buffer = io.BytesIO()
|
||||
Image.new("RGB", (8, 8), color).save(buffer, format="PNG")
|
||||
raw = buffer.getvalue()
|
||||
encoded = base64.b64encode(raw).decode("ascii")
|
||||
output = nbformat.v4.new_output(
|
||||
"display_data", data={"image/png": encoded},
|
||||
metadata={"filename": "figure"} if same_stem else {},
|
||||
)
|
||||
notebook = nbformat.v4.new_notebook(cells=[
|
||||
nbformat.v4.new_code_cell(f"# Notebook {index}", outputs=[output]),
|
||||
nbformat.v4.new_markdown_cell(
|
||||
"",
|
||||
attachments={"picture.png": {"image/png": encoded}},
|
||||
),
|
||||
])
|
||||
nbformat.write(notebook, source)
|
||||
sources.append(source)
|
||||
expected.append(raw)
|
||||
output_dir = root / "converted"
|
||||
result = subprocess.run(
|
||||
[sys.executable, str(SCRIPTS_DIR / "source_to_md/doc_to_md.py"),
|
||||
*map(str, sources), "-o", str(output_dir)],
|
||||
capture_output=True, text=True, encoding="utf-8", check=False,
|
||||
)
|
||||
self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
|
||||
markdown_paths = sorted(output_dir.glob("*.md"))
|
||||
self.assertEqual(len(markdown_paths), 2)
|
||||
self.assertEqual(list(output_dir.glob("*.png")), [])
|
||||
for markdown, raw in zip(markdown_paths, expected):
|
||||
refs = re.findall(r"!\[[^\]]*\]\(([^)]+)\)", markdown.read_text(encoding="utf-8"))
|
||||
self.assertEqual(len(refs), 2)
|
||||
for ref in refs:
|
||||
self.assertEqual(Path(ref).parent, Path(f"{markdown.stem}_files"))
|
||||
self.assertEqual((output_dir / ref).read_bytes(), raw)
|
||||
profile = json.loads(markdown.with_suffix(".conversion_profile.json").read_text())
|
||||
self.assertEqual(profile["outputs"]["image_count"], 2)
|
||||
manifest = json.loads((output_dir / profile["outputs"]["image_manifest"]).read_text())
|
||||
self.assertEqual(len(manifest), 2)
|
||||
manager = ProjectManager(base_dir=root / "projects")
|
||||
with redirect_stdout(io.StringIO()):
|
||||
project = Path(manager.init_project("notebooks", quick_generate=True))
|
||||
summary = manager.import_sources(str(project), list(map(str, markdown_paths)), copy=True)
|
||||
self.assertEqual(summary["skipped"], [])
|
||||
self.assertEqual(len(summary["assets"]), 2)
|
||||
for path, raw in zip(summary["markdown"], expected):
|
||||
markdown = Path(path)
|
||||
refs = re.findall(r"!\[[^\]]*\]\(([^)]+)\)", markdown.read_text(encoding="utf-8"))
|
||||
for ref in refs:
|
||||
self.assertEqual((markdown.parent / ref).read_bytes(), raw)
|
||||
manifest = json.loads((project / "images/image_manifest.json").read_text())
|
||||
self.assertEqual({row["source_namespace"] for row in manifest},
|
||||
{path.stem for path in markdown_paths})
|
||||
self.assertEqual({(project / "images" / row["filename"]).read_bytes() for row in manifest},
|
||||
set(expected))
|
||||
|
||||
|
||||
class DocxContentLossTests(unittest.TestCase):
|
||||
def _convert(self, document, patch_archive=None):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
|
||||
+43
@@ -24,6 +24,49 @@ import excel_to_md # noqa: E402
|
||||
|
||||
|
||||
class ExcelContentLossTests(unittest.TestCase):
|
||||
def test_same_stem_batch_preserves_each_workbook_and_profile(self) -> None:
|
||||
for directory_input in (False, True):
|
||||
for separate_output in (False, True):
|
||||
with self.subTest(directory=directory_input, output=separate_output):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
root = Path(tmp)
|
||||
sources = []
|
||||
totals = {"budget.xlsx": 20000, "budget.xlsm": 22500}
|
||||
for name, total in totals.items():
|
||||
workbook = Workbook()
|
||||
workbook.active.append(["Category", "Amount"])
|
||||
workbook.active.append(["Total", total])
|
||||
source = root / name
|
||||
workbook.save(source)
|
||||
workbook.close()
|
||||
sources.append(source)
|
||||
output = root / "output" if separate_output else root
|
||||
arguments = [root] if directory_input else sources
|
||||
if separate_output:
|
||||
arguments += ["-o", output]
|
||||
result = subprocess.run(
|
||||
[sys.executable, str(SCRIPTS_DIR / "source_to_md/excel_to_md.py"),
|
||||
*map(str, arguments)],
|
||||
capture_output=True, text=True, encoding="utf-8", check=False,
|
||||
)
|
||||
self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
|
||||
self.assertIn("Success: 2/2, Failed: 0", result.stderr)
|
||||
self.assertIn("[INFO] Renamed output for", result.stderr)
|
||||
self.assertEqual(sorted(p.name for p in output.glob("*.md")),
|
||||
["budget.md", "budget_2.md"])
|
||||
profiles = sorted(output.glob("*.conversion_profile.json"))
|
||||
self.assertEqual(len(profiles), 2)
|
||||
seen = set()
|
||||
for path in profiles:
|
||||
profile = json.loads(path.read_text(encoding="utf-8"))
|
||||
source_name = profile["source"]["name"]
|
||||
seen.add(source_name)
|
||||
markdown = output / profile["outputs"]["markdown"]
|
||||
self.assertIn(f"| Total | {totals[source_name]} |",
|
||||
markdown.read_text(encoding="utf-8"))
|
||||
self.assertEqual(path, markdown.with_suffix(".conversion_profile.json"))
|
||||
self.assertEqual(seen, set(totals))
|
||||
|
||||
def _convert(self, workbook, **options):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
source = Path(tmp) / "book.xlsx"
|
||||
|
||||
@@ -2,8 +2,8 @@
|
||||
"sourceId": "shadcn",
|
||||
"repo": "https://github.com/shadcn-ui/ui.git",
|
||||
"ref": "main",
|
||||
"commit": "c2a67849be260701852b0977ba15eb5f3a0ce2a4",
|
||||
"commit": "02cad61166c4c20edbc8a3f71bd720eed4c1f49b",
|
||||
"adapter": "claude-skill",
|
||||
"sourcePath": "skills/shadcn",
|
||||
"syncedAt": "2026-10-10T16:00:00Z"
|
||||
"syncedAt": "2026-10-11T16:00:00Z"
|
||||
}
|
||||
@@ -2,8 +2,8 @@
|
||||
"sourceId": "ui-ux-pro-max",
|
||||
"repo": "https://github.com/nextlevelbuilder/ui-ux-pro-max-skill.git",
|
||||
"ref": "main",
|
||||
"commit": "50d8a7de0900119855614541f15a1a616691eb33",
|
||||
"commit": "67803eac81607188194aea9842a723b89326d90d",
|
||||
"adapter": "claude-skill",
|
||||
"sourcePath": ".claude/skills/ui-ux-pro-max",
|
||||
"syncedAt": "2026-10-09T10:26:29Z"
|
||||
"syncedAt": "2026-10-11T16:00:00Z"
|
||||
}
|
||||
Reference in new issue
Block a user