Wikiopisto

Bottien käyttö Wikimedioissa/Python botin ohjelmoiminen

< Bottien käyttö Wikimedioissa

Tästä alunperin lähteestä suoraan kopioidusta materiaalista muokataan tänne parhaillaan vain python kielellä ohjelmoituja botteja koskeva tiivistelmä.

Tämä teksti on tuotu vieraskielisestä lähteestä ja sen käännös on keskeneräinen.
Voit auttaa Wikikirjastoa tekemällä käännöksen loppuun.

Robotit eli botit ovat automaattisia prosesseja, jotka ovat wikipedian kanssa vuorovaikutuksessa samalla tavoin kuin wikipediaa muokkaavat ihmiset. Tässä artikkelissa kerrotaan kuinka Wikipediabotteja voidaan ohjelmoida python-kielellä.

Mihin botteja tarvitaan?

muokkaa

Boteilla voidaan automatisoida monia rutiineja ja ne suoritavat niitä paljon nopeammin kuin ihmiset. Jos tehtävänä esimerkiksi olisi lisätä Malline:Malline 1000 sivua sisältävä luokan joka sivulle, sopii se paljon paremmin botille kuin ihmiselle.

Pitääkö ne ohjelmoida itse?

muokkaa

Lukijan, jollla ei entuudestaan ole lainkaan ohjelmointikokemusta, on varmasti helpompi käyttää jonkun toisen kehittämää bottia kuin ohjelmoida se itse. Valmita botteja voi hakea sivulta Wikipedia:Botit ja ellei sieltä löydy sopivaa voi toiminnallisesti spesifioidun bottin ohjelmointipyynnös esittää sivulla Wikipedia:Botit/pyynnöt. Yksi mahdollisuus on myös käyttää ja/tai muokata ns. AWB-boteja (semi-bots).

Vaikka tämän oppaan lähdedokumentti on suunnattu lähinnä jo jonkinverran ohjelmistokokemusta omaaville lukijoille, yritetään tämä pelkästään python kieleen keskittyvä tivistelmä muokata sellaiseksi, ettei sen lukeminen edellytä mitään esitietoja ohjelmoinnista. Tämän pitäisi periaateessa olla mahdollista, koska python-kieltä suositellaan usein ensimmäiseksi ohjelmointikieleksi.

Oppaan tarkoituksena on opettaa lukijalle sekä ohjelmoinnin perustaidot python kielellä että bottiohjelmointiin perusteet siihen liittyviä ohjelmointiesimerkkejä käyttäen. Lopuksi esitetyt asiat kootaan yhteen jonkun mahdollisimman yksinkertaisen toimivan botiesimerkin avulla. Opas ei kuitenkaan pyri käsittelemään sen enempää python kieltä kuin bottiohjelmointiakaan tyhjentävästi, vaan toimiatoimia starttioppaana, joka helpottaa aihepiiriin liittyvän tietämyksen syventämistä oppaan loppuun linkkitettyjen muiden oppaiden ja ohjeiden avulla.

Miten Wikipediabotti toimii?

muokkaa

Yleistä

muokkaa
 

Juuri samoin kuin tavallinen käyttäjä myös botti voi lukea wikipediasisältöjä, tehdä niihin liittyviä päätelmiä ja editoida niitä. Tietokoneet ovat joissakin tehtävissä ihmistä parempia ja toisissa huomompia. Ne kykenevät lukemaan ja kirjoittamaan useita sivuja sekunnissa, mutta ne eivät ole kovin hyviä tekemään sivujen sisältöjen arviointtiin perustuvia päätöksiä.

Sivuja selaimella editoivan ihmisen tavoin myös botti suorittaa Wikipediaan sivujen HTTP-hakukomentoja, tekee laskelmia ja päätelmiä serverin palauttamasta datasta ja suorittaa päätelmien tulosten perusteella lisähakuja.

Esimerkiksi listan linkeistä, jotka viittaavat tietylle sivulle botti saa kertomalla serverille, että "haluan Special:Whatlinkshere". Serveri vastaa palauttamalla generoimansa HTML-datan ja botti ohjaa toimintaansa serverilta saamansa datan sisällön mukaisesti.

APIs for bots

muokkaa

An API is an w:en:Application Programming Interface. It describes how a bot should interact with wikipedia.

The most common API is probably w:en:Screen scraping. Screenscraping involves requesting a wikipedia page, and then looking at the raw HTML (just like a human could do by clicking View->Source in most browsers) and then extracting certain values from the HTML. It is not recommended, as interface may change at any moment, and it's a huge server load. However currently this is the only way to get an HTML-formatted wiki page. Also currently scraping the edit page and making a http post submission based on the data from the edit form is the only way to perform edits.

You can pass action=render to index.php to reduce the amount of data to transfer and the dependence to the user interface. Other parameters of index.php may be useful: Manual:Parameters to index.php is a partial list of these parameters and their possible values.

Other APIs that may be of use:

  • MediaWiki API extension - this is the next generation API that may one day be expanded to include data posting, as well as various data requests. Currently a number of features have already been implemented, such as revision+content, log events, watchlist, and many other. Data is available in many different formats (w:en:JSON,w:en:XML,w:en:YAML,...). See it live. Features are being ported from the older Query API interface.
    Status: Partially complete engine feature, available on all Wikimedia projects.
  • Query API -- a multi-format API to query data directly from the Wiki-servers (Live)
    Status: Production. This is an extension enabled on all WikiMedia servers.
  • Special:Export feature (bulk export of xml formatted data); see Parameters_to_Special:Export for arguments;
    Status: Production. Built-in engine feature, available on all WikiMedia servers.
  • Raw page: passing action=raw or action=raw&templates=expand to index.php allows for a direct access to the source of pages

The Wikipedia web servers are configured to grant requests for compressed (w:en:gzip) content. This can be exploited by including a line "Accept-Encoding: gzip" in the HTTP request header; if the HTTP reply header contains "Content-Encoding: gzip", the document is in gzip form; otherwise, it is in the regular uncompressed form; not checking the returned header may lead to incorrect results. Note that this is specific to the web server and not to the MediaWiki software; other sites employing MediaWiki may not have this feature.

Logging in as your bot when making edits

muokkaa

It will be necessary for an approved bot to be logged in under its user account before making edits. A bot is free to make read requests without logging in, but ideally it should log in for all activities. One can login via a POST at a URL like http://en.wikipedia.org/w/index.php?title=Special:Userlogin&action=submitlogin&type=login passing the POST data wpName=BOTUSERNAME&wpPassword=BOTPASSWORD&wpRemember=1&wpLoginattempt=Log+in

Logging in is also possible using API, using an URL like http://en.wikipedia.org/w/api.php?action=login&lgname=BOTUSERNAME&lgpassword=BOTPASSWORD. For security reasons, it is safer to use a URL like http://en.wikipedia.org/w/api.php?action=login to POST lgname=BOTUSERNAME&lgpassword=BOTPASSWORD

Once logged in, the bot will need to save Wikipedia cookies and make sure it passes these back when making edit requests. When editing on the English Wikipedia, these three cookies should be used: enwikiUserID, enwikiToken, and enwikiUserName. enwiki_session is required to actually make an edit (or commit some change), or else this message will appear.

See w:en:Hypertext Transfer Protocol and w:en:HTTP cookie for more details.

Edit tokens

muokkaa

Wikipedia uses a system of edit tokens for making edits to wikipedia pages and performing some other operations.

What this means is that it is not possible for your bot to pass a single POST request via HTTP to make a page edit. Just like a human editor, it is necessary for the bot to go through a number of stages:

Why am I being returned an empty or near-empty edit token?

muokkaa

Your bot may be returned an empty edit token or the edit token "+\". This indicates that the bot is not logged in; this may be due to a failure in its authentication with the server, or a failure in storing and returning the correct cookies.

Edit conflicts

muokkaa

A high possibility is that your bot will get caught in an edit conflict, in which another user will have made an edit between your requesting an edit (and getting your edit token) and actually making the edit. Another possible conflict is an edit/delete conflict, where the edited page is deleted between when the edit form is downloaded and submitted.

An edit conflict error may incorrectly result if the value of the wpEdittime variable in the data sent to the server is wrong. In a similar way, a wrong value of wpStarttime may result in an edit/delete conflict error.

To check for edit conflicts, sadly you will have to do it by checking the HTML document. Generally, if the server returns a "200 OK" HTML header when submitting a form, the edit has not been done (either because of a conflict, a loss of session token, protection, or a database locks). On a successful edit, the server returns a "302 Moved Temporarily", and the page URL in the Location field.

Overview of the process of developing a bot

muokkaa

Actually coding or writing a bot is only one part of developing a bot. You should generally follow the development cycle below. Failure to comply with this development cycle, particularly the sections on Wikipedia bot policy, may lead to your bot failing to be approved or being blocked from editing Wikipedia.

 
Overview of Wikipedia bot development cycle

Software Elements Analysis:

muokkaa
  • The first task in creating a Wikipedia bot is extracting the requirements or coming up with an idea. If you don't have an idea of what to write a bot for, you could pick up ideas at requests for work to be done by a bot.
  • Make sure an existing bot isn't already doing what you think your bot should do. To see what tasks are already being performed by a bot, see the list of currently operating bots.

Specification:

muokkaa
  • Specification is the task of precisely describing the software to be written, possibly in a rigorous way. You should come up with a detailed proposal of what you want it to do. Try to discuss this proposal with some editors and refine it based on feedback. Even a great idea can be made better by incorporating ideas from other editors.
  • In the most basic form, your specified bot must meet the following criteria:
  • The bot is harmless (it must not make edits that could be considered vandalism)
  • The bot is useful (it provides a useful service more effectively than a human editor could), and
  • The bot does not waste server resources.

w:en:Software architecture:

muokkaa
  • Think about how you might create it and which programming language and tools you would use. Architecture is concerned with making sure the software system will meet the requirements of the product as well as ensuring that future requirements can be addressed. There are different types of bots and the main body of the article below will cover this technical side.

Implementation:

muokkaa

Implementation (or coding) involves reducing design to code. It may be the most obvious part of the software engineering job but it is not necessarily the largest portion. In the implementation stage you should:

  • Create a user page for your bot. Your bot's edits must not be made under your own account. Your bot will need its own account with its own username and password.
  • Add these details to your proposal and post it to requests for bot approval
  • Add the same information to the user page of the bot. You should also add a link to the approval page (whether approved or not) for each function. People will comment on your proposal and it will be either accepted or rejected.
  • Code your bot in your chosen programming language.

Testing:

muokkaa

If accepted, it would probably be put on a trial period during which it may be run to fine-tune it and iron out any bugs. You should test your bot widely and ensure that it works correctly. At the end of the trial period it would hopefully be accepted.

Documentation:

muokkaa

An important (and often overlooked) task is documenting the internal design of your bot for the purpose of future maintenance and enhancement. This is especially important if you are going to allow clones of your bot. Ideally, you should post up the source code of your bot on its userpage if you want others to be able to run clones of it. This code should be well documented for ease of use.

Software Training and Support:

muokkaa

You should be ready to field queries or objections to your bot on your user talk page.

Maintenance:

muokkaa

Maintaining and enhancing your bot to cope with newly discovered problems or new requirements can take far more time than the initial development of the software. Not only may it be necessary to add code that does not fit the original design but just determining how software works at some point after it is completed may require significant effort.

  • If you want to make a major functionality change to your bot in the future, you should request this as above using the requests for bot approval.

Considerations about type of bot to be developed

muokkaa
  • Is it to run w:en:server-side or w:en:client-side?
  • Is it to be manually assisted or fully automated?
  • Will its requests be logged?
  • Will it be reporting its actions to a human?

General guidelines for running a bot

muokkaa

In addition to the official bot policy, which covers the main points to consider when developing your bot, there are a number of more general advisory points to consider when developing your bot.

Bot best practice

muokkaa
  • Try not to make more than 10 requests (read and write added together) per minute.
  • Try to run the bot only at low server load times, or throttle the read/write request rate down at busy server times.
  • Edit/write requests are more expensive in server time than read requests. Be edit-light.
  • Do not make multi-threaded requests. Wait for one server request to complete before beginning another
  • Back off upon receiving errors from the server. Errors are an indication often of heavy server load. Try to respond by backing off and not repeatedly hammering the server with replacement requests.
  • Try to consolidate edits. One single large edit is better than 10 smaller ones. Try and write your code with this in mind.
  • Make use of the Assert Edit extension, an extension explicitly designed for bots to check certain conditions, which is enabled on Wikipedia.

Common bot features you should consider implementing

muokkaa

Timers

muokkaa

You don't want your bot to edit too fast. Timers are ways for bots to control how fast they edit. A common way to do this is to send your bot to "sleep" for a certain time after performing an action before doing anything else. In Perl, this is accomplished by the simple command sleep(10); where 10 is the number of seconds to sleep for. In C# you can use Thread.Sleep(10000); which will make the bot sleep for 10 seconds. Make sure to call using System.Threading first.

Manual assistance

muokkaa

If your bot is doing anything that requires judgement or evaluation of context (e.g., correcting spelling) then you should consider making your bot manually-assisted. That is, not making edits without human confirmation.

Disabling the bot

muokkaa

It is good bot policy to have a feature to disable the bot's operation if it is requested. You should probably have the bot refuse to run if a message has been left on its talk page, on the assumption that the message may be a complaint against its activities. This can be checked by looking for the "You have new messages..." banner in the HTML for the edit form. Remember that if your bot goes bad, it is your responsibility to clean up after it! You can also have a page that will turn the bot off if True on the page is changed. This can be done by grabbing and checking the page before each edit.

Signature

muokkaa

Just like a human, if your bot makes edits to a talk page on wikipedia, it should sign its post with four tildes (~~~~). It should not sign any edits to text in the main namespace.

What technique and language should I use?

muokkaa

Semi-Bots or auxiliary software

muokkaa

In addition to true bots, there are a number of semi-bots known as auxiliary software available to anyone. Most of these take the form of enhanced web browsers with Wikipedia-specific functionality. The most popular one of these is AWB (Click here for a full list of these and other editing tools).

Bot clones

muokkaa

There are already a number of bots running on wikipedia. Many of these bots publish their w:en:source code and allow you to subscribe to them, download and operate a copy (clone) of them in order to perform useful tasks whilst browsing wikipedia.

Developing a new bot

muokkaa

Bots are essentially small computer applications or applets written in one or more programming languages. There are generally two types of bots: w:en:client-side and w:en:server-side. Client-side bots run on your local machine, and can only be run by you. Server-side bots are hosted online by your own machine or on a remote webserver, and (if you chose) can be initiated by others.

An overview is given below of various languages that can be used for writing a wikipedia bot. For each, a list is given of external articles advising how to get started programming in that language, as well as a list of existing libraries (sub-programs) you can use with your bot to prevent having to "reinvent the wheel" for basic functions.

Python

muokkaa

Introduction to use

An example of some Python code: 

def addt5(x):
    return x+5

def dotwrite(ast):
    nodename = getNodeName()
    label=symbol.sym_name.get(int(ast[0]),ast[0])
    print ' %s [label="%s' % (nodename, label) ,
    if isinstance(ast[1], str):
        if ast[1].strip():
            print '= %s "];' % ast[1]
        else:
            print '"]'
    else:
        print '"]'
        children = []
        for n, child in enumerate(ast[1:]):
            children.append(dotwrite(child))
        print ' %s -> {' % nodename,
        for name in children:
            print '%s' % name,

Getting started with Python:

Libraries:

Muut kielet

muokkaa
w:en:Perl
Perl has a w:en:run-time compiler. This means that it is not necessary to compile builds of your code yourself as it is with other programming languages.
w:en:PHP
w:en:PHP can also be used for programming bots. PHP is an especially good choice if you wish to provide a webform-based interface to your bot.
w:en:Microsoft .NET
Languages include C#, Managed C++, w:en:Visual Basic .NET, J#, w:en:JScript .NET, w:en:IronPython, and w:en:Windows PowerShell.
Java
Generally developed with Eclipse
Ruby
RWikiBot is a Ruby framework for writing bots. Currently, it is under development and looking for contributors. It uses MediaWiki's official API, and as such is limited in certain capabilities.
Noudettu kohteesta ”https://fi.wikiversity.org/w/index.php?title=Bottien_käyttö_Wikimedioissa/Python_botin_ohjelmoiminen&oldid=3246”