Install psalm as dev, sync scripts updates

This commit is contained in:
Clemens Schwaighofer
2023-03-09 16:27:10 +09:00
parent 6bec59e387
commit feba79a2e8
2099 changed files with 283333 additions and 32 deletions

View File

@@ -0,0 +1,62 @@
# Adding a new issue type
To add a new issue type there are a number of required steps, listed below.
## Generating a new shortcode
Run `bin/max_used_shortcode.php` and note the value it printed (`$max_shortcode`)
## Create issue class
Create a class in `Psalm\Issue` namespace like this:
```php
<?php
namespace Psalm\Issue;
final class MyNewIssue extends CodeIssue
{
public const SHORTCODE = 123;
public const ERROR_LEVEL = 2;
}
```
For `SHORTCODE` value use `$max_shortcode + 1`. To choose appropriate error level see [Error levels](../running_psalm/error_levels.md).
There a number of abstract classes you can extend:
* `CodeIssue` - non specific, default issue. It's a base class for all issues.
* `ClassIssue` - issue related to a specific class (also interface, trait, enum). These issues can be suppressed for specific classes in `psalm.xml` by using `referencedClass` attribute
* `PropertyIssue` - issue related to a specific property. Can be targeted by using `referencedProperty` in `psalm.xml`
* `FunctionIssue` - issue related to a specific function. Can be suppressed with `referencedFunction` attribute.
* `ArgumentIssue` - issue related to a specific argument. Can be targeted with `referencedFunction` attribute.
* `MethodIssue` - issue related to a specific method. Can be targeted with `referencedMethod` attribute.
* `ClassConstantIssue` - issue related ot a specific class constant. Can be targeted with `referencedConstant`.
* `VariableIssue` - issue for a specific variable. Targeted with `referencedVariable`
## Add a `config.xsd` entry
All issue types needs to be listed in `config.xsd`, which is used to validate `psalm.xml`. Choose appropriate `type` attribute. E.g. for issues extending `PropertyIssue` use `type="PropertyIssueHandlerType"`.
## Add a doc page for your new issue
Every issue needs to be documented. Create a markdown file in `docs/running_psalm/issues` folder. Make sure to include a snippet of code illustrating your issue. Important: snippets must use fenced php code block and must include opening PHP tag (`<?php`). The snippet must actually produce the issue you're documenting. It's checked by our test suite.
## Add links to the doc page
Add links to the doc page you created to `docs/running_psalm/error_levels.md` and `docs/running_psalm/issues.md`
## Run documentation tests
```
$ vendor/bin/phpunit tests/DocumentationTest.php
```
It will check whether you did all (or at least most) of the steps above.
## Use your new issue type in Psalm core
```php
IssueBuffer::maybeAdd(new MyNewIssue(...))
```

View File

@@ -0,0 +1,81 @@
# Altering callmaps
## Intro
One of the first things most contributors start with is proposing changes to
callmaps.
Callmap is a data file (formatted as a PHP file returning an array) that tells
Psalm what arguments function/method takes and what it returns.
There are two full callmaps (`CallMap.php` and `CallMap_historical.php`) in
`dictionaries` folder, and a number of delta files that provide information on
how signatures changed in various PHP versions. `CallMap_historical` has
signatures as they were in PHP 7.0, `CallMap.php` contains current signatures
(for PHP 8.1 at the time of writing).
## Full callmap format
Full callmaps (`CallMap.php` and `CallMap_historical.php`) have function/method
names as keys and an array representing the corresponding signature as a value.
First element of that value is a return type (it also doesn't have a key), and
subsequent elements represent function/method parameters. Parameter name for an
optional parameter is postfixed with `=`.
## Delta file format
Delta files (named `CallMap_<PHP major version><PHP minor version>_delta.php`)
list changes that happened in the corresponding PHP version. There are
three section with self-explanatory names: `added` (for functions/methods that
were added in that PHP version), `removed` (for those that were removed) and
`changed`.
Entry format for `removed` and `added` section matches that of a full callmap,
while `changed` entries list `old` and `new` signatures.
## How Psalm uses delta files
When the current PHP version is set to something other than the latest PHP
version supported by Psalm, it needs to process delta files to arrive at a
version of callmap matching the one that is used during analysis. Psalm uses
the following process to do that:
1. Read `CallMap.php` (Note: it's the one having the latest signatures).
2. If it matches configured PHP version, use it.
3. If the callmap delta for previous PHP version exists, read that.
4. Take previous callmap delta and apply it in reverse order. That is, entries
in `removed` section are added, those in `added` section are removed and
`changed.new` signatures in the current callamp are replaced with
`changed.old`.
5. Goto 2
## Consistent histories
To make sure there are no mismatches in deltas and the callmap, CI validates
that all function/method entries have consistent histories. E.g. that the
signature in `changed.new` matches the one in `CallMap.php`, the `removed`
entries are actually absent from `CallMap.php` and so on.
## Typical changes
To put that into practical perspective, let's see how a couple of typical
callmap changes may look like.
### Adding a new function
Say, there's a function added in PHP 8.1, e.g. `array_is_list()`. Add it to the
`CallMap_81_delta.php` (as it was introduced in PHP 8.1), and `CallMap.php` (as
it exists in the latest PHP version). Here's [the PR that does it](https://github.com/vimeo/psalm/pull/6398/files).
### Correcting the function signature
Assume you found an incorrect signature, the one that was always different to what
we currently have in Psalm. This will need a change to `CallMap_historical.php`
(as the signature was always that way) and `CallMap.php` (as the signature is
still valid). Here's [the PR that does it](https://github.com/vimeo/psalm/pull/6359/files).
If function signature is correct for an older version but has changed since you
will need to edit the delta for PHP version where signature changed and
`CallMap.php` (as this new signature is still valid). Here's
[the PR that does it (makes `timestamp` nullable)](https://github.com/vimeo/psalm/pull/6244/files).

View File

@@ -0,0 +1,73 @@
# How Psalm works
The entry point for all analysis is [`ProjectAnalyzer`](https://github.com/vimeo/psalm/blob/master/src/Psalm/Internal/Analyzer/ProjectAnalyzer.php)
`ProjectAnalyzer` is in charge of two things: Scanning and Analysis
## Scanning
For any file or set of files, Psalm needs to determine all the possible dependencies and get their function signatures and constants, so that the analysis phase can be done multi-threaded.
Scanning happens in `Psalm\Internal\Codebase\Scanner`.
The first task is to convert a file into a set of [PHP Parser](https://github.com/nikic/PHP-Parser) statements. PHP Parser converts PHP code into an abstract syntax tree that Psalm uses for all its analysis.
### Deep scanning vs shallow scanning
Psalm then uses a custom PHP Parser `NodeVisitor` called [`ReflectorVisitor`](https://github.com/vimeo/psalm/blob/master/src/Psalm/Internal/PhpVisitor/ReflectorVisitor.php) which has two modes when scanning a given file: a shallow scan, where it just gets function signatures, return types, constants, and inheritance, or a deep scan, where it drills into every function statement to get those dependencies too (e.g. the class names instantiated by a function). It only does a deep scan on files that it knows will be analysed later (so the vast majority of the vendor directory, for example, just gets a shallow scan).
So, when analysing the `src` directory, Psalm will deep scan the following file:
src/A.php
```php
<?php
use Vendor\VendorClass;
use Vendor\OtherVendorClass;
class A extends VendorClass
{
public function foo(OtherVendorClass $c): void {}
}
```
And will also deep scan the file belonging to `Vendor\VendorClass`, because it may have to check instantiations of properties at some point.
It will do a shallow scan of `Vendor\OtherVendorClass` (and any dependents) because all it cares about are the method signatures and return types of the variable `$c`.
### Finding files from class names
To figure out the `ClassName` => `src/FileName.php` mapping it uses reflection for project files and the Composer classmap for vendor files.
### Storing data from scanning
For each file that `ReflectorVisitor` visits, Psalm creates a [`FileStorage`](https://github.com/vimeo/psalm/blob/master/src/Psalm/Storage/FileStorage.php) instance, along with [`ClassLikeStorage`](https://github.com/vimeo/psalm/blob/master/src/Psalm/Storage/ClassLikeStorage.php) and [`FunctionLikeStorage`](https://github.com/vimeo/psalm/blob/master/src/Psalm/Storage/FunctionLikeStorage.php) instances depending on the file contents.
Once we have a set of all files and their classes and function signatures, we calculate inheritance for everything in the [`Populator`](https://github.com/vimeo/psalm/blob/master/src/Psalm/Internal/Codebase/Populator.php) class and then move onto analysis.
At the end of the scanning step we have populated all the necessary information in [`ClassLikes`](https://github.com/vimeo/psalm/blob/master/src/Psalm/Internal/Codebase/ClassLikes.php), [`Functions`](https://github.com/vimeo/psalm/blob/master/src/Psalm/Internal/Codebase/Functions.php) and [`Methods`](https://github.com/vimeo/psalm/blob/master/src/Psalm/Internal/Codebase/Methods.php) classes, and created a complete list of `FileStorage` and `ClassLikeStorage` objects (in [`FileStorageProvider`](https://github.com/vimeo/psalm/blob/master/src/Psalm/Internal/Provider/FileStorageProvider.php) and [`ClassLikeStorageProvider`](https://github.com/vimeo/psalm/blob/master/src/Psalm/Internal/Provider/ClassLikeStorageProvider.php) respectively) for all classes and files used in our project.
## Analysis
We analyse files in [`FileAnalyzer`](https://github.com/vimeo/psalm/blob/master/src/Psalm/Internal/Analyzer/FileAnalyzer.php)
The `FileAnalyzer` takes a given file and looks for a set of top-level components: classes, traits, interfaces, functions. It can look inside namespaces and extract the classes, interfaces, traits and functions in them as well.
It delegates the analysis of those components to [`ClassAnalyzer`](https://github.com/vimeo/psalm/blob/master/src/Psalm/Internal/Analyzer/ClassAnalyzer.php), [`InterfaceAnalyzer`](https://github.com/vimeo/psalm/blob/master/src/Psalm/Internal/Analyzer/InterfaceAnalyzer.php) and [`FunctionAnalyzer`](https://github.com/vimeo/psalm/blob/master/src/Psalm/Internal/Analyzer/FunctionAnalyzer.php).
Because its the most basic use case for the line-by-line analysis (no class inheritance to worry about), lets drill down into `FunctionAnalyzer`.
### Function Analysis
`FunctionAnalyzer::analyze` is defined in [`FunctionLikeAnalyzer`](https://github.com/vimeo/psalm/blob/master/src/Psalm/Internal/Analyzer/FunctionLikeAnalyzer.php). That method first gets the `FunctionLikeStorage` object for the given function that we created in our scanning step. That `FunctionLikeStorage` object has information about function parameters, which we then feed into a [`Context`](https://github.com/vimeo/psalm/blob/master/src/Psalm/Context.php) object. The `Context` contains all the type information we know about variables & properties (stored in `Context::$vars_in_scope`) and also a whole bunch of other information that can change depending on assignments and assertions.
Somewhere in `FunctionLikeAnalyzer::analyze` we create a new [`StatementsAnalyzer`](https://github.com/vimeo/psalm/blob/master/src/Psalm/Internal/Analyzer/StatementsAnalyzer.php) and then call its `analyze()` method, passing in a set of PhpParser nodes. `StatementAnalyzer::analyze` passes off to a number of different checkers (`IfAnalyzer`, `ForeachAnalyzer`, `ExpressionAnalyzer` etc.) for more thorough analysis.
At each line the `Context` object may or may not be manipulated. At branching points (if statements, loops, ternary etc) the `Context` object is cloned and then, at the end of the branch, Psalm figures out how to resolve the changes and update the uncloned `Context` object.
The `NodeDataProvider` stores a type for each PhpParser node.
After all the statements have been analysed we gather up all the return types and compare them to the given return type.
### Type Reconciliation
While some updates to the `Context` object are straightforward, others are not. Updating the `Context` object in the light of new type information happens in [`Reconciler`](https://github.com/vimeo/psalm/blob/master/src/Psalm/Type/Reconciler.php), which takes an array assertions e.g. `[“$a” => “!null”]` and a list of existing type information e.g. `$a => string|null` and return a set of updated information e.g. `$a => string`

View File

@@ -0,0 +1,40 @@
# Contributing to Psalm
Psalm is made possible through the contributions of [hundreds of developers](https://github.com/vimeo/psalm/graphs/contributors).
Hopefully you can be one of them?
## Getting started
[Heres a rough guide to the codebase](how_psalm_works.md).
[Here's the philosophy underpinning the Psalms development](philosophy.md).
I've also put together [a list of Psalms complexities](what_makes_psalm_complicated.md).
Are you looking for low-hanging fruit? Here are some [GitHub issues](https://github.com/vimeo/psalm/issues?q=is%3Aissue+is%3Aopen+label%3A%22easy+problems%22) that shouldn't be too difficult to resolve.
### Dont be afraid!
One great thing about working on Psalm is that its _very_ hard to introduce any sort of type error in Psalms codebase. There are almost 5,000 PHPUnit tests, so the risk of you messing up (without the CI system noticing) is very small.
### Why static analysis is cool
Day-to-day PHP programming involves solving concrete problems, but they're rarely very complex. Psalm, on the other hand, attempts to solve a pretty hard collection of problems, which then allows it to detect a ton of bugs in PHP code without actually executing that code.
There's a lot of interesting theory behind the things Psalm does, too. If you want you can go very deep, though you don't need to know really any theory to improve Psalm.
Lastly, working to improve static analysis tools will also make you a better PHP developer it'll help you think more about how values flow through your program.
### Guides
* [Editing callmaps](editing_callmaps.md)
* [Adding a new issue type](adding_issues.md)
## Pull Requests
Before you send a pull request, make sure you follow these guidelines:
Run integration checks locally: `composer tests`
If you're adding new features or fixing bugs, dont forget to add tests!

View File

@@ -0,0 +1,70 @@
# Philosophy
This is about why Psalm is the way it is. This is a living document!
## Psalm is a tool for PHP written in PHP
### Advantages
- PHP is fast enough for most use-cases
- Writing the tool in PHP guarantees that PHP community members can contribute to it without too much difficulty
### Drawbacks
- Psalm is slow in very large monorepos (over 5 million lines of checked-in code).
- Psalms language server is more limited in what it can provide than comparable compiled-language tools. For example, it's infeasible to find/change all occurrences of a symbol, in all files that use it, as you type.
### Comparison to other languages & tools
Many languages have typecheckers/compilers written in the same language. Popular examples include Go, Rust, and TypeScript. Python is a special case where the semi-official typechecker [MyPy](https://github.com/python/mypy) (written in Python) can also [compile to a C Python extension](https://github.com/python/mypy/blame/master/mypyc/README.md#L6-L10), which runs 4x faster than the interpreted equivalent.
Some interpreted languages have unofficial open-source typecheckers written in faster compiled languages. In all cases a single mid-to-large company is behind each effort, with a small number of contributors not employed by that company:
- PHP
- [NoVerify](https://github.com/VKCOM/noverify) written in Go. Runs much faster than Psalm (but does not support many modern PHP features)
- Ruby
- [Sorbet](https://sorbet.org/) - written in C
- Python
- [Pyre](https://github.com/facebook/pyre-check) - written in OCaml
- [Hack](https://github.com/facebook/hhvm) - the typechecker is written in OCaml and Rust
## Psalm's primary purpose is finding bugs
Psalm does a lot, but people mainly use it to find potential bugs in their code.
All other functionality the language server, security analysis, manipulating/fixing code is a secondary concern.
## It's designed to be run on syntactically-correct code
Psalm is almost always run on PHP code that parses a lint check (`php -l <filename>`) i.e. syntactically-correct code. Psalm is not a replacement for that syntax check.
Given Psalm is almost always used on syntatically-correct code it should use a parser built for that purpose, and `nikic/php-parser` is the gold-standard.
Where Psalm needs to run on syntactically-incorrect code (e.g. in language server mode) Psalm should still use the same parser (and work around any issues that it produces).
## Docblock annotations are better than type-providing plugins
Psalm offers a plugin API that allows you to tell it what about your program's property types, return types and parameter types.
Psalm aims to operate in a space with other static analysis tools. The more users rely on plugins, the less chance those other tools have to understand the user's intent.
Psalm should encourage developers to use docblock annotations rather than type-providing plugins. This was a driving force in the adoption of [Conditional Types](../annotating_code/type_syntax/conditional_types.md) which allowed Psalm to replace some of its own internal type-providing plugins.
The other benefit to docblock annotations is verifiability for the most part Psalm is able to verify that docblock annotations are correct, but it cannot provide many assurances when plugins are used.
This doesnt mean that plugins as a whole are bad, or that they cant provide useful types. A great adjacent use of plugins is to provide stubs with Psalm type annotations for libraries that dont have them. This helps the PHP ecosystem because those stubs may eventually make their way into the project currently being stubbed.
## Docblock annotations that can be verified are better than those that cannot
Psalm currently supports a number of function/class docblock annotations that it's unable to verify:
- `@psalm-assert`, `@psalm-assert-if-true`, `@psalm-assert-if-false`
- `@property`, `@method`, `@mixin`
Whenever new docblock annotations are added, effort should be made to allow Psalm to verify their correctness.
## In certain circumstances docblock annotations are better than PHP 8 attributes
For information that's just designed to be consumed by static analysis tools, docblocks are a better home than PHP 8 attributes.
A rationale is provided in [this article](https://psalm.dev/articles/php-8-attributes).

View File

@@ -0,0 +1,82 @@
# Things that make developing Psalm complicated
This is a somewhat informal list that might aid others.
## Type inference
Type inference is one of the big things Psalm does. It tries to figure out what different PHP elements (function calls, if/for/foreach statements etc.) mean for the data in your code.
Within type inference there are a number of tricky areas:
#### Loops
Loops are hard to reason about - break and continue are a pain. This analysis mainly takes place in `LoopAnalyzer`
#### Combining types
There are lots of edge-cases when combining types together, given the many types Psalm supports. Type combining occurs in `TypeCombiner`.
#### Logical assertions
What effect do different PHP elements have on user-asserted logic in if conditionals, ternarys etc. This logic is spread between a number of different classes.
#### Generics & Templated code
Figuring out how templated code should work (`@template` tags) and how much it should work like it does in other languages (Hack, TypeScript etc.) is tricky. Psalm also supports things like nested templates (`@template T1 of T2`) which makes things trickier
## Detecting dead code
Detecting unused variables requires some fun [data-flow analysis](https://psalm.dev/articles/better-unused-variable-detection).
Detecting unused classes and methods between different runs requires maintaining references to those classes in cache (see below).
## Supporting the community
- **Supporting formal PHPDoc annotations**
- **Supporting informal PHPDoc annotations**
e.g. `ArrayIterator|string[]` to denote an `ArrayIterator` over strings
- **non-Composer projects**
e.g. WordPress
## Making Psalm fast
#### Parser-based reflection
Requires scanning everything necessary for analysis
#### Forking processes** (non-windows)
Mostly handled by code borrowed from Phan, but can introduce subtle issues, also requires to think about how to make work happen in processes
#### Caching thing
see below
## Cache invalidation
#### Invalidating analysis results
Requires tracking what methods/properties are used in what other files, and invalidating those results when linked methods change
#### Partial parsing
Reparsing bits of files that have changed, which is hard
## Language Server Support
#### Handling temporary file changes
When files change Psalm figures out what's changed within them to avoid re-analysing things unnecessarily
#### Dealing with malformed PHP code
When people write code, it's not always pretty as they write it. A language server needs to deal with that bad code somehow
## Fixing code with Psalter
#### Adding/replacing code
Figuring out what changed, making edits that could have been made by a human
#### Minimal diffs
hard to change more than you need