View source for Parsoid

{{original|mw:{{PAGENAME}}}}
<onlyinclude>{{Wikimedia engineering project information
| name        = Parsoid
| logo        = Parsoid word logo.svg
| description = A bidirectional runtime wikitext parser. Converts back and forth between wikitext and HTML/XML DOM with RDFa.
| start       = 2011-10
| group       = [[mw:Content Transform Team]]
| lead        = [[mw:User:Ssastry|Subramanya Sastry]]
| team        = [[mw:User:Arlolra|Arlo Breault]], [[mw:User:cscott|C.Scott Ananian]], [[mw:User:IHurbainPalatin_(WMF)|Isabelle Hurbain-Palatin]], [[mw:User:MSantos_(WMF)|Mateus Santos]], [[mw:User:SBailey_(WMF)|Shannon Bailey]], [[mw:User:JGiannelos_(WMF)|Yiannis Giannelos]]
| EPM         = Sérgio Lopes
| projectpage = Parsoid
| Phabricator = parsoid
| updates = {{Mwll|mw:Parsoid/Releases}}
| progress = {{Mwll|mw:Parsoid/Deployments}}
}}</onlyinclude>

[[File:Parsoid HTML-RDFa content model.svg|framed|Artist's impression of the Parsoid HTML5 + RDFa wiki runtime]]

'''Parsoid'''는 [[mw:wikitext|위키텍스트]]와 HTML 사이를 앞으로 및 뒤로 변환하는 것을 허용하는 라이브러리입니다. 원래 응용프로그램은 자바스크립트 (Node.js를 사용)로 작성되었고 2012년 12월에 위키미디어 클러스터에서 작동했었습니다. 2019년에, Parsoid가 PHP로 포팅되었고, PHP 버전이 2019년 12월에 위키미디어 클러스터의 JS 버전을 대체했습니다. Parsoid는 궁극적으로 미디어위키의 현재 네이티브 파서를 대체하는 것을 목표로 [[mw:Parsing/Parser Unification|핵심 미디어위키에 통합될 것]]입니다.

Parsoid (PHP 버전)는 2020년 9월에 출시된 미디어위키 1.35에 [[mw:Parsoid/PHP|기본적으로 번들로 제공]]됩니다. 비-위키미디어 설치에 대해, Parsoid/JS는 [[mw:Version lifecycle|2021년 9월]]에서 미디어위키 1.31 (LTS)의 수명이 종료 때까지 지원됩니다.

==Technical details==
Parsoid는 미디어위키의 [[mw:wikitext|위키텍스트]] 구문과 동등한 HTML/RDFa 문서 모델 사이를 앞으로 및 뒤로 번역할 수 있는 응용프로그램으로 자동화된 처리와 풍부한 편집을 위한 향상된 지원을 제공합니다.

그것은 2012년부터 위키미디어 재판에서 한 팀에 의해 계속 개발 중에 있습니다. 그것은 현재 [[Extension:VisualEditor|시각 편집기 확장]], [[Extension:Flow|토론 확장]], [[mw:Content translation|내용 번역]]과 [[mw:Parsoid/Users|기타 응용 프로그램]]에서 광범위하게 사용됩니다.

Parsoid는 흠없는 앞으로-및-뒤로 변환을 제공하기 위해, 즉, 정보 손실을 방지하고 역시 "더러운 차이(dirty diffs)"를 방지하기 위해 의도됩니다.

위키미디어 위키에서, 여러 응용프로그램에 대해, Parsoid는 현재 Parsoid에 의해 번역된 HTML을 저장하는 [[RESTBase]] 뒤에 프록시됩니다. RESTBase는 결국 미디어위키와 더 긴밀하게 통합된 캐시로 [[phab:T250500|대체될 것]]으로 예상됩니다.

전체 프로젝트의 자세한 내용에 대해, 2013년 3월로부터 [http://blog.wikimedia.org/2013/03/04/parsoid-how-wikipedia-catches-up-with-the-web/ 이 블로그 게시물]을 참조하십시오. 사용 중인 HTML 모델에 대해 읽기 위해, [[mw:Parsoid/MediaWiki DOM spec|미디어위키 DOM 스펙]]을 참조하십시오.

Parsoid는 원래 웹 서비스로 구조화되었고 자바스크립트에서 작성되었으며, Node.js의 사용을 만듭니다. [https://www.youtube.com/watch?v=lQGfuLP9MqA2019년 2019년 2월 기술 토론] ([[c:File:The_Long_And_Winding_Road_To_Making_Parsoid_The_Default_MediaWiki_Parser.pdf|발표 자료]])과 [[phab:phame/post/view/189/parsoid_in_php_or_there_and_back_again/|블로그 게시물]]에서 포팅 과정을 설명합니다. Parsoid 확장 API는 현재 활발하게 개발 중입니다; [[Mw:Wikimedia Technical Talks#Episode%206:%20Retargeting%20extensions%20to%20work%20with%20Parsoid|2020년 8월의 기술 토론]]은 이 작업을 설명합니다.

'''GitHub Repository:''' https://github.com/wikimedia/parsoid

== Usage ==
* [[mw:Parsoid/Releases|Parsoid/Releases]] - List of releases made for parsoid
* [[mw:Parsoid/API|Parsoid/API]] - for the web API
* [[mw:Specs/HTML|MediaWiki DOM spec]] - to make sense of the HTML that you get from the API, designed to be useful as a future storage format
* [[mw:Parsoid/LanguageConverter|Parsoid/LanguageConverter]] - notes on Parsoid's implementation of [[mw:LanguageConverter|LanguageConverter]]
* [[mw:Parsoid/Known differences with PHP parser output|Parsoid/Known differences with PHP parser output]]

== Installation ==
'''미디어위키 1.35 LTS'''에서, ''Parsoid/PHP''는 번들로 포함되고 시각 편집기에서 자동적으로 로딩됩니다. 단일 서버에서 사용되면 설정이 요구되지 않습니다.

만약 현재 개발 중인 '''미디어위키 1.36''' 버전을 사용하려면, 2020년 8월 24일부터 Parsoid는 명시적으로 로딩이 되어야 합니다 (자동-로딩 핵킹은  1.36-wmf.6에서 [[gerrit:c/mediawiki/extensions/VisualEditor/+/622187|제거되었습니다]]). 다음을 LocalSettings.php에 추가하십시오:
<pre>
wfLoadExtension( 'Parsoid', __DIR__ . 'vendor/wikimedia/parsoid/extension.json' );
</pre>
이것은 1.36의 출시에 대해 변경될 것으로 예상됩니다.

== Development ==

Development happens in the {{git repo|project=mediawiki/services/parsoid|text=Parsoid Git repository}}.
Code review happens in [[gerrit:q/project:mediawiki/services/parsoid|Gerrit]].
See [[Mw:Gerrit/Getting started|Gerrit/Getting started]] to set up an account for yourself.

If you use the [[Special:MyLanguage/MediaWiki-Vagrant|MediaWiki-Vagrant]] development environment using a virtual machine, you can simply add the role <code>visualeditor</code> to it and it will set up a working Parsoid along with [[Extension:VisualEditor]].
(This may have been broken by the switch to Parsoid/PHP: [[phab:T258940|T258940]])

Note that the most-recently released version of Parsoid is written in PHP, and installation of [[Mw:Parsoid/PHP|Parsoid/PHP]] is what is described below.
This is what you should use if you are running MediaWiki 1.35 or later.
Check [[Mw:Parsoid/JS|Parsoid/JS]] if you are running the old version of Parsoid written in JavaScript, and used for MW 1.34 and earlier.

=== Linking a developer checkout of Parsoid ===

In a standard MediaWiki installation, Parsoid code is bundled in two different ways: first, Parsoid is included from MediaWiki as a composer library, <code>wikimedia/parsoid</code>.
This contains the main codebase, but does not contain the REST API used by VisualEditor and RESTBase.
In order to enable the REST API, the extension code included in the Parsoid library can be loaded with a call to <code>wfLoadExtension(...)</code> in your <code>LocalSettings.php</code>.

For development purposes you usually want to use a git checkout of Parsoid, and not the version bundled in MediaWiki core as a composer library.
The following lines added to <code>LocalSettings.php</code> allow use of a git checkout of Parsoid (optionally), load the Parsoid REST API with <code>wfLoadExtension</code> (rather than using the version bundled in VisualEditor) and manually do the Parsoid configuration which is usually done by VisualEditor:

<syntaxhighlight lang="php">
$parsoidInstallDir = 'vendor/wikimedia/parsoid'; # bundled copy
#$parsoidInstallDir = '/my/path/to/git/checkout/of/Parsoid';

// For developers: ensure Parsoid is executed from $parsoidInstallDir,
// (not the version included in mediawiki-core by default)
// Must occur *before* wfLoadExtension()
if ( $parsoidInstallDir !== 'vendor/wikimedia/parsoid' ) {
    // AutoLoader::registerNamespaces was added in MW 1.39
    AutoLoader::registerNamespaces( [
        // Keep this in sync with the "autoload" clause in
        // $parsoidInstallDir/composer.json
        'Wikimedia\\Parsoid\\' => "$parsoidInstallDir/src",
    ] );
}

wfLoadExtension( 'Parsoid', __DIR__ . "$parsoidInstallDir/extension.json" );

# Manually configure Parsoid
$wgVisualEditorParsoidAutoConfig = false;
$wgParsoidSettings = [
    'useSelser' => true,
    'rtTestMode' => false,
    'linting' => false,
];
$wgVirtualRestConfig['modules']['parsoid'] = [
    // URL to the Parsoid instance.
    // If Parsoid is not running locally, you should change $wgServer to match the non-local host 
    // While using Docker in macOS, you may need to replace $wgServer with http://host.docker.internal:8080
    // While using Docker in linux, you may need to replace $wgServer with http://172.17.0.1:8080
    'url' => $wgServer . $wgScriptPath . '/rest.php',
    // Parsoid "domain", see below (optional, rarely needed)
    // 'domain' => 'localhost',
];
unset( $parsoidInstallDir );
</syntaxhighlight>

These lines are not necessary for most users of VisualEditor, who can use auto-configuration and the bundled Parsoid code included in MediaWiki 1.35 and VisualEditor, but they will be required for most developers.

If you're serving MediaWiki with Nginx, you'll need to also add something like this in your server block (Assuming your MediaWiki setup has its files residing in <code>/w/</code>):

<syntaxhighlight lang="nginx">
location /w/rest.php/ {
    try_files $uri $uri/ /w/rest.php?$query_string;
}
</syntaxhighlight>

To test proper configuration, visit <code>{$wgScriptPath}/rest.php/{$domain}/v3/page/html/Main%20Page</code> where {{phpi|$domain}} is the hostname in your {{phpi|$wgCanonicalServer}}.
(Note that production WMF servers do not expose the Parsoid REST API to the external network.)

=== Running the tests ===

To run all parser tests and mocha tests:

<syntaxhighlight lang="shell-session">
$ composer test
</syntaxhighlight>

The parser tests have quite a few options now which can be listed using <code>php bin/parserTests.php --help</code>.

If you have the environment variable <code>MW_INSTALL_DIR</code> pointing to a configured MediaWiki installation, you can run some additional tests with:

<syntaxhighlight lang="shell-session">
$ composer phan-integrated
</syntaxhighlight>

=== Converting simple wikitext ===

You can convert simple wikitext snippets from the command line using the <code>parse.php</code> script in the <code>bin/</code> directory:

 echo '[[Foo]]' | php bin/parse.php

The parse script has a lot of options.
<kbd><code>php bin/parse.php --help</code></kbd> gives you information about this.

===Debugging Parsoid (for developers)===

See [[Mw:Parsoid/Debugging|Parsoid/Debugging]] for debugging tips.

== Continuous Integration ==
''As of October 2021''

Parsoid is always available as a library since it is a composer dependency of MediaWiki core. But two pieces are not enabled:

* Parsoid ServiceWiring
* Parsoid's external REST api

The test runner [[Mw:Quibble|Quibble]] would enable it if it detects <code>mediawiki/services/parsoid.git</code> has been cloned as part of the build. In which case it:

* points the autoloader for <code>Wikimedia\Parsoid</code> to the cloned code (effectively replacing the version installed by composer)
* Load the extension <code>wfLoadExtension( 'Parsoid', '/path/to/cloned/repo' );</code>

The ServiceWiring should be enabled in MediaWiki starting with 1.38.

The REST API would theorically never get merged in MediaWiki: a) it has never been exposed to the public in production, it is an internal API used by RESTBase which is going away; b) it never has been security audited and c) it is redundant with the enterprise MediaWiki API. The solution will be for VisualEditor to invoke Parsoid directly via the VisualEditor Action API which would save a round trip through the REST API.

Loading the extension is thus a hack which enables using interfaces subject to change and which we don't really want people to use yet.

For most purposes, parsoid should thus not be added as a CI dependency, the only exception as of October 2021 is the Disambiguator MediaWiki extension.

Loading parsoid as an extension let us run MediaWiki integration test jobs against <code>mediawiki/services/parsoid.git</code> (such as Quibble, apitesting) and ensure Parsoid and MediaWiki work together.

An extension may be able to write tests with Parsoid even when the repository has not been cloned. Since it is a composer dependency of MediaWiki core the <code>MediaWiki\Parsoid</code> namespace is available, but the service wiring part is not (it is <code>extension/src</code> in the Parsoid repository and exposed as the <code>\MWParsoid</code> namespace). The <code>ParsoidTestFileSuite.php</code> code would only run the parser tests if Parsoid has been loaded (which should be the default with MediaWiki 1.38).

For CI, Parsoid is tested against the tip of mediawiki, whereas mediawiki is tested with the composer dependency. In case of a breaking change, the Parsoid change get merged first (which breaks its CI but not MediaWiki one) and MediaWiki get adjusted when Parsoid is updated. It is thus a one way change.

== Release build ==
For MediaWiki release builds, we have an integration of Parsoid ServiceWiring into VisualEditor in order to have VisualEditor work without further configuration (beside a <code>wfLoadExtension( 'VisualEditor' )</code>). The release build also enables the REST API and hook everything us so that parsoid works out of the box. This is done by copying a bit of parsoid code into VisualEditor which is not in the master branch of VisualEditor since that would be obsolete as soon as Parsoid is updated. Instead the code is maintained in two places.

== Technical documents ==
* [[Mw:Parsoid/Internals|Parsoid/Internals]]: documentation about Parsoid internals with links to other details.
*[[Mw:Parsing/Notes/Moving Parsoid Into Core/Porting|PHP Porting notes]] and [[Mw:Parsoid/PHP/Help wanted|help-wanted tasks]]
* [[Mw:Parsoid/Deployments|Parsoid deployment agenda on Wikimedia cluster]] (code normally deployed every Monday and Wednesday between 1pm - 1:30pm PST)
* [[Mw:Parsoid/Round-trip testing|Parsoid/Round-trip testing]]: The round-trip testing setup we are using to test the wikitext -> HTML DOM -> wikitext round-trip on actual Wikipedia content.
* [[Mw:Parsoid/Visual Diffs Testing|Parsoid/Visual Diffs Testing]]: Info about visual diff testing for comparing Parsoid's html rendering with php parser's html rendering + a testreduce setup for doing mass visual diff tests.
* [[Mw:Parsoid/limitations|Parsoid/limitations]]: Limitations in Parsoid, mainly contrived templating (ab)uses that don't matter in practice. Could be extended to be similar to [[meta:MNPP|the preprocessor upgrade notes]] ('''Might need updating''')
* [[Mw:Parsoid/Bibliography|Parsoid/Bibliography]]: Bibliography of related literature

=== Links for Parsoid developers ===

* See [[Mw:Parsoid/Debugging|Parsoid/Debugging]] for debugging tips.
* [[Mw:Parsoid/Upgrade packages|Upgrading or adding packages]] to Parsoid
* [https://github.com/wikimedia/parsoid/blob/39ea4e49783b1f88326d6ebddd14e08bb39cb649/tools/sync-parserTests.js#L5-L34 See these instructions] for syncing Parsoid's copy of parser tests to/from core
* Parsoid has a limited [https://doc.wikimedia.org/Parsoid/master/#!/api/Parsoid library interface] for invoking it programatically.
* [https://www.youtube.com/watch?v=lS1xPkERWCM Tech Talk about Retargeting extensions to work with Parsoid]
* [[Mw:Parsoid/So you want your extension to work with Parsoid|So you want your extension to work with Parsoid]]
* [[Mw:Parsoid/Parsoid HTML Specification Versioning|Parsoid HTML Specification Versioning]]
* [[Mw:Parsoid/So you are going to change Parsoid output|So you are going to change Parsoid output]]

=== Links for Parsoid deployers (to the Wikimedia cluster) ===

* [[Mw:Parsoid/Deployments|Parsoid/Deployments]]
* [http://parsoid-rt-tests.wikimedia.org/commits RT testing commits] (useful to check regressions and fixes)
* [[Mw:Labsconsole:Parsoid#Deploying%20changes|Deployment instructions for Parsoid]]
* [https://logstash.wikimedia.org/app/kibana#/dashboard/parsoid Kibana dashboard]
* [http://grafana.wikimedia.org/dashboard/db/parsoid-timing-wt2html Grafana dashboard for wt2html metrics]
* [http://grafana.wikimedia.org/dashboard/db/parsoid-timing-html2wt Grafana dashboard for html2wt metrics]
* [https://grafana.wikimedia.org/dashboard/db/parsoid-http-status-codes Grafana dashboard for non-200 responses]
*[https://grafana.wikimedia.org/d/000000607/cluster-overview?orgId=1&from=now-24h&to=now&var-datasource=eqiad%20prometheus%2Fops&var-cluster=parsoid&var-instance=All Prometheus breakdown for the Parsoid cluster on eqiad]
*[https://grafana.wikimedia.org/d/000000607/cluster-overview?orgId=1&from=now-24h&to=now&var-datasource=codfw%20prometheus%2Fops&var-cluster=parsoid&var-instance=All Prometheus breakdown for the Parsoid cluster on codfw]
* [[Mw:CI/JJB|Jenkins Job Builder docs]] for updating jenkins jobs

==See slso==
* [https://en.wikipedia.org/wiki/API API]
* [[RESTBase]]: a caching / storing API proxy for page HTML translated by Parsoid
* Quarterly review meetings of the Parsoid team: [[m:WMF Metrics and activities meetings/Quarterly reviews/Parsoid, Services, MW Core, Ops, RelEng, MM, Labs and ECT, April 2015|April 2015]], [[m:WMF Metrics and activities meetings/Quarterly reviews/Services/January 2015|January 2015]] ([[m:WMF Metrics and activities meetings/Quarterly reviews|earlier]])
* [[mw:Future/Parser plan]]: Early (now relatively old) design ideas and issues
* [[mw:Special:PrefixIndex/Parsoid/]]: Parsoid-related pages on this wiki
* [[Extension:ParsoidBatchAPI]]
* [https://github.com/wikimedia/parsoid-jsapi parsoid-jsapi]: a high-level interface for extraction and transformation of wikitext, similar to the [https://github.com/earwig/mwparserfromhell mwparserfromhell] API.
* [[mw:Alternative parsers]]

== External links ==

* [[git:mediawiki/services/parsoid/|Source code]] ([https://github.com/wikimedia/parsoid/ GitHub mirror])
* [https://doc.wikimedia.org/Parsoid/master/ JS Documentation] (old version of Parsoid)
* [https://doc.wikimedia.org/Parsoid-PHP/master/ PHP Documentation]
* [[commons:Category:Parsoid|''Parsoid'' on the Wikimedia Commons]]

[[Category:Software]]